Paul Cerrato, M.A.

home

authors

paul-cerrato-ma

, Siddhartha Mukherjee, M.D., called cancer the “Emperor of All Maladies.” The term is spot on. The disease commands the attention of all who come in contact with it, whether they be patients, clinicians, or technologists, claiming the lives of more than 8 million worldwide and disrupting the lives of millions more. That’s not to suggest that we aren’t making any progress. The latest statistics from the 

 state that the U.S. cancer death rate has dropped by 27% in 25 years. But that still means about 1.7 million new cancer cases and over 606,000 deaths are expected this year alone. Many stakeholders are hoping that advances in artificial intelligence and machine learning will make a significant dent in these numbers.

, we discussed the value of machine-learning algorithms to assist in the diagnosis of melanoma and to improve colorectal cancer screening. The software used to help detect skin cancer takes advantage of neural networks while the colorectal screening system, commercially available as ColonFlag in Europe and LGI Flag in the United States, uses a machine-learning tool called extreme gradient boosting. There have been similar advances in several other specialties, including breast and lung cancers.

To make sense of the role of ML in breast cancer risk assessment, it helps to understand how breast density fits into the picture. Women with denser breasts, which contain more fibrous and glandular tissue, are at greater risk of the malignancy than women with fatty breasts. Traditionally, there have been tewo risk models to help clinicians in this area: the Gail model and, more recently, the Tyrer-Cuzick (TC) model. The latter has been incorporated into the Gail model to take into consideration a woman’s breast density. The TC model is now considered the clinical standard. However, because assessing a patient’s breast density can be subjective, it may be possible to improve the assessment using deep learning to analyze mammograms for subtle differences not detectable by the naked eye. 

, a Ph.D. candidate from the Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, and his colleagues have developed an algorithm that looks at full-field mammography images to assist clinicians in 

Yala and his colleagues used data from patients’ EHRs and questionnaires to evaluate their likelihood of developing breast cancer using 3 different models. One used traditional risk factors, including the TC model and logistic regression, the second used a convolutional neural network (CNN) to analyze mammogram images, and the third combined both approaches. The analysis found that the CNN was much better than the TC model for predicting breast cancer; combining CNN with TC was even better. One of the surprising findings was that women who did not have dense breasts but were considered at high risk based on this more sophisticated risk assessment model were almost four times as likely to have developed cancer, when compared to those with dense breasts and model-assessed low risk.

Machine learning is also helping pathologists as they try to interpret biopsy slides and determine whether or not a patient’s breast cancer has metastasized to nearby lymph nodes. 

 from Google AI Healthcare, and associates applied an algorithm called Lymph Node Assistant (LYNA) to two separate data sets of pathology slide images. The first set of slides was also reviewed by one of two pathologists while slides from the second data set were interpreted by two pathologists. Comparing LYNA to the skills of a practicing pathologist, investigators found an area under the curve (AUC) of 99.3% for nodal metastasis present or absent for the first data set. (AUC, also called receiver operating characteristic (ROC) is a metric that helps to distinguish between two diagnostic groups, in this case slides that indicate metastasis vs no metastasis). Pathologists returned an AUC of 96.6% when they were allowed to take as much time as they needed to arrive at a diagnosis. That AUC dropped to 81% when a real-world scenario was put in place: They were only allowed one minute per slide. Liu and associates concluded that: “Artificial intelligence algorithms can exhaustively evaluate every tissue patch on a slide, achieving higher tumor-level sensitivity than, and comparable slide level performance to, pathologists. These techniques may improve the pathologist’s productivity and reduce the number of false negatives associated with morphologic detection of tumor cells.” In a separate study that used LYNA, 

 demonstrated that combining the algorithm with the services of a human pathologist resulted in more accurate detection of micrometastases than either the pathologists on their own or LYNA on its own.

One of the challenges facing pathologists is their lack of consistency in making a cancer diagnosis. ML—enhanced algorithms may eventually solve this problem by supplementing pathologists’ judgment. 

 addressed this challenge using a convolutional neural network to improve the diagnosis of invasive breast cancer in whole slide images. Invasive disease was defined as the spread of the tumor beyond the breast’s milk ducts or lobules. The researchers used a ConvNet classifier and validated it with images from three different institutions. They then evaluated it using pathologic and normal cases from the Cancer Genome Atlas and University Hospitals Case Medical Center in Cleveland. When compared to pathologists’ findings, the classifier generated a positive predictive value of 71.6% and a negative predictive value of 96.8%. One of the criticisms of ML—based diagnostic tools is that they are often trained on a single data set and may not be generalizable to the larger patient population. The research team addressed this concern by using 2 different data sets. Comparing the two resulted in highly correlated performance measures (r >/–0.8).

Technologists are hoping that AI-based diagnostic tools like this will help address the epidemic of diagnostic errors we face in clinical medicine. It’s estimated that a delayed diagnosis is one of the four common causes of diagnostic errors, which also include missed diagnosis, misdiagnosis, i.e. incorrectly diagnosed disease, and overdiagnosis. Unfortunately, technology can’t solve all these problems. A report from the Institute for Health Improvement and CRICO, the risk management foundation of the Harvard Medical Institutions, points out that there are over 100 million outpatient referrals to specialists annually, but up to half of these referrals are never completed. A single missed referral can easily lead to a delayed diagnosis and a malpractice claim. 

: “Of malpractice claims related to missed or delayed diagnosis in the ambulatory setting, almost half involve failure to follow up, many of which involve problems with specialist referrals.”

Luke Sato, M.D., chief medical officer at CRICO, explains that the diagnostic errors that lead to malpractice claims fall into two broad categories: cognitive errors and system-based errors. Although diagnostic errors may result from the failure of a physician to order a colonoscopy, for instance, or refer a patient to the appropriate specialist, CRICO has found that errors also result from failing to “close the loop,” for example, a fault in the healthcare system that somehow interferes with the referral process. How serious is this problem? Sato points out that although 

 say that they include the patient’s history and the reason for referring patients to a specialist “always” or “most of the time,” fewer than 35% of specialists say they receive this information. In some cases, the technology is unavailable within the EHR to allow clinical or administrative staff to properly close the loop. In other cases, it is available but not well utilized. The IHI/CRICO white paper discusses a long list of communication breakdowns responsible for the problem, but part of the problem is a lack of accountability. In the U.S. healthcare ecosystem, there is no one consistently held responsible for scheduling the process. Equally troubling is the fact that no one is consistently in charge of knowledge management.

Tackling Cervical Cancer with Deep Learning

While diagnostic errors remain a stubborn, expensive problem, there are many success stories worth telling. The prevention of cervical cancer in the United States is one of the genuine triumphs in oncology.  In the 1940s, the disease was a major cause of death among women of childbearing age in the U.S., but since the Pap smear was introduced in the 1950s, invasive cervical cancer has dropped dramatically: From 1955 to 1992, its incidence and death rate dropped by more than 60%, Unfortunately, the disease continues to be a major cause of death and suffering in poorer countries that can’t afford routine Pap smear screening. Worldwide, there are about 500,000 cases, and 

 occur in low and middle income countries. A recent data analysis suggests that an automated visual evaluation of cervical images, developed with the help of a convolutional neural network, can have a significant impact on this worldwide problem. (Two video tutorials are available that provide a basic understanding of how neural networks work. One was created for an episode of PBS’s 

Currently, clinicians in these poorer countries use an acetic acid test to estimate if a woman has precancer of the cervix. The test involves applying acetic acid to the cervix to see if the tissues turn white, which suggests precancer or cancer. While the test is inexpensive, its interpretation is subjective, and isn’t very accurate in differentiating between precancer and more common minor abnormalities. That in turn results in overtreatment and undertreatment. 

 Ph.D., with the Intellectual Ventures Global Good Fund, Bellevue, Washington, and his colleagues, analyzed a data set of cervical images from over 9,000 women in Costa Rica who had been screened for the malignancy and followed for seven years.

The automated visual evaluation algorithm was designed to accomplish two tasks:  It located the cervix on the camera image, and it predicted the likelihood of the patient having cervical intraepithelial neoplasia (CIN2+), a group of precancerous lesions. Hu and his colleagues found that the machine-learning enabled algorithm was more accurate than manually performed visual inspection of the cervix and more accurate than Pap smears.

Radiologists have two invaluable tools to help them detect diagnostic clues in the images they review: Their eyes and their clinical training.  But the human eye obviously has its limitations.  There are millions of pixels in a typical X-ray, making it virtually impossible to notice subtle abnormalities that can be easily detected by a machine-learning based algorithm. The computer’s advantage has been observed in numerous diagnostic settings, including lung cancer.

Although screening of high-risk patients with computed tomography (CT scans) has been shown to reduce the death rate in lung cancer patients, many cancers are still missed due to misjudgements in reading and interpreting the scans. Several software systems have been used to help address this problem, including those that rely on deep learning neural networks. 

, a Ph.D. candidate with the Department of Systems & Technology, Auburn University in Alabama, and his colleagues, have created a two-tier system to analyze CT scans, looking for suspicious pulmonary nodules that suggest lung cancer. One neural network was used to locate nodule candidates worth investigating and the second network helped eliminate false positive readings of the CT scans. Gruetzemacher and his associates used a data set derived from National Cancer Institute’s Cancer Imaging Archive. They explain in their report: “Deep learning offers unique advantages over traditional methods for nodule detection. Rather than relying on mathematical techniques or hand-crafted features, deep learning enables learning internal feature representations directly from the input data.” Put another way, neural networks teach themselves how to recognize nodules by evaluating thousands of images. Their two-tiered system worked with a detection rate of 89.3%, with only 1.7 false positives per scan. 

Machine learning will never replace an experienced clinician’s diagnostic skills, but as these research projects demonstrate, it is slowly emerging as a valuable complement that will likely save lives.

Paul Cerrato has more than 30 years of experience working in healthcare as a clinician, educator, and medical editor. He has written extensively on clinical medicine, electronic health records, protected health information security, practice management, and clinical decision support. He has served as editor of Information Week Healthcare, executive editor of Contemporary OB/GYN, senior editor of RN Magazine, and contributing writer/editor for the Yale University School of Medicine, the American Academy of Pediatrics, Information Week, Medscape, Healthcare Finance News, IMedicalapps.com, and Medpage Today. HIMSS has listed Mr. Cerrato as one of the most influential columnists in healthcare IT.

John D. Halamka, M.D., leads innovation for Beth Israel Lahey Health. Previously, he served for over 20 years as the chief information officer (CIO) at the Beth Israel Deaconess Healthcare System. He is chairman of the New England Healthcare Exchange Network (NEHEN) and a practicing emergency physician. He is also the International Healthcare Innovation professor at Harvard Medical School. As a Harvard professor, he has served the George W. Bush administration, the Obama administration and national governments throughout the world, planning their healthcare IT strategies. In his role at BIDMC, Dr. Halamka was responsible for all clinical, financial, administrative and academic information technology, serving 3,000 doctors, 12,000 employees, and 1 million patients.

 Get the best insights in digital health 

Replacing the Old School with New-School AI in Medicine

Mental Health Apps: Do They Work? Are They Safe?

Tackling the Misdiagnosis Epidemic with Human and Artificial Intelligence

Articles

Stakeholders hope AI, machine learning will significantly decrease death toll.

Will AI Improve Cancer Diagnosis and Treatment?

Software as a medical device (SaMD) and U.S. Food and Drug Administration (FDA) 510(k) clearance might not be the most familiar terms to all health system leaders, physicians and nurses who use computers, but if they hope to take advantage of the latest artificial intelligence (AI)-enabled programs, they will need to keep up. The most recent FDA proposed regulatory framework on SaMD is likely to transform patient care in the next few years. Understanding how the guidelines fit into the picture will enable clinicians and their organizations to take full advantage of the machine learning-based diagnostic and treatment algorithms now being introduced.

To make sense of SaMD, it helps to understand the FDA’s expanded definition of a medical device. In the not-too-distant past, a medical device was a physical object — an EKG machine or a blood pressure cuff, for example. But the term now encompasses software used to assist in the diagnosis, treatment, mitigation or prevention of disease. As we explained in 

The Transformative Power of Mobile Medicine

, the FDA believes that most mobile apps intended to diagnose or treat disease are “medical devices under federal statute.” On the other hand, apps or other types of software that are either designed to provide wellness services or that receive, store, transmit and offer more general health information are not considered medical devices.

Software that the agency says falls into the SaMD category includes:

Mobile apps that transform a mobile platform into a regulated medical device and therefore are mobile medical apps: These mobile apps use a mobile platform’s built-in features such as light, vibrations, camera or other similar sources to perform medical device functions (e.g., mobile medical apps that are used by a licensed practitioner to diagnose or treat a disease).

Mobile apps that connect to an existing device type for purposes of controlling its operation, function or energy source and therefore are mobile medical apps: These mobile apps are those that control the operation or function (e.g., change settings) of an implantable or body worn medical device.

Mobile apps that are used in active patient monitoring or analyzing patient-specific medical device data from a connected device and therefore are mobile medical apps.

Although the FDA’s position on SaMD was designed to protect the public from unscrupulous vendors who create ineffective or unsafe digital tools, since publishing these regulations, the agency has come to realize that they might be too restrictive in some situations and could actually damper healthcare innovation. That’s especially the case when it comes to machine learning-based software. A closer look at the FDA clearance/approval process helps explain why this problem has arisen.

Medical device applications are handled with three processes:

 seeks to gain clearance of a device that is at least as safe and effective as a legally marketed device that’s already available to the public. 

, on the other hand, is a more stringent review process required of Class III medical devices, which support or sustain human life or “which present a potential, unreasonable risk of illness or injury.” De Novo classification applies to novel medical devices “for which general controls alone, or general and special controls, provide reasonable assurance of safety and effectiveness for the intended use, but for which there is no legally marketed predicate device.” Unfortunately, AI and machine-learning algorithms do not easily fit into any of these 3 classifications, which has forced FDA to develop a new approach, which it has published as 

Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)

The need for new regulations can be summed up in two words: adaptive technology. The current regulations were designed to handle “locked” technology, such as software whose functioning does not change. Because many machine-learning systems are self-learning, by definition the software does change, adapting to new input from training and test data sets. In 

, these software systems “have the potential to adapt and optimize device performance in real-time to continuously improve healthcare for patients.”

To cope with this new situation, the agency is proposing what it calls a “predetermined change control plan” for device developers who are seeking premarket clearance. Submitters would have to provide FDA with a plan that spells out the types of anticipated modifications they see coming down the road, as well as the methodology they plan to use to implement these changes in a controlled manner. The ultimate goal of such a plan is to manage patients’ risk from the software. Vendors who agree to this proposed arrangement would be expected to commit to transparent, real-world performance monitoring of their AI and ML software. They would also be willing to provide periodic updates to the agency. According to FDA, this new approach to SaMD would allow the regulator to embrace the iterative improvement power of AI- and machine learning-based software as a medical device, while assuring patient safety.

To understand how the proposed regulatory framework world work in the real world, one first needs to grasp two rather obscure terms that the agency has introduced: SaMD pre-specifications (SPS) and algorithm change protocol (ACP). SPS refers to the 

 that a vendor plans to implement in its software. In other words, the company has to spell out what its algorithm is supposed to become as it learns from its new inputs, namely the new training and testing data that it encounters. ACP describes the 

 the vendor plans to use to make sure the revised algorithm performs as expected. The guidelines explain: “The ACP is a step-by-step delineation of the data and procedures to be followed so that the modification achieves its goals and the device remains safe and effective after the modification.” The agency envisions four components to this change protocol, as outlined in the Figure.

These proposals might sound rather vague without a real-world example of how they might apply to an actual AI/ML system. Imagine a mobile app that allows a smartphone to take a photo of a skin lesion so that a physician can make a clinical judgement about whether it’s a normal mole or skin cancer. Since the app drives clinical management of a serious healthcare problem, it’s considered a medical device that falls within the FDA’s jurisdiction. If a vendor decides it wants to modify the app, it first has to explain what changes it plans to make — the SPS portion of the review process. The FDA framework document gives possible modifications. For instance, the algorithm may be exposed to a new data set with images of skin lesions with the goal of improving its sensitivity and specificity. A more sensitive algorithm is more likely to detect true positive findings, such as how the number of photos identified as skin cancer that really 

 skin cancer would increase as a result of the new training. The software would be more specific if it succeeded in identifying more true negatives, such as how the number of photos labeled as not skin cancer that really aren’t skin cancer would increase.

The change protocol or ACP for this dermatologic app would involve the manufacturer submitting to FDA its method for making these improvements. The developer would have to explain how it generated the data set, discuss reference standard labeling and provide a comparative analysis and performance requirements, including sensitivity and specificity. If the vendor could collect the needed data and demonstrate its improved sensitivity and specificity, and it updated its labeling and communicated the improvements to its audience, “the modified algorithm that ‘learned’ on real-world data can be marketed without additional FDA review.”

On the other hand, if this same hypothetical app manufacturer wanted to sell this mobile app to patients to use on their smartphones, it would face a different regulatory scenario. The vendor would direct patients to follow up with a dermatologist after receiving a preliminary analysis of a malignancy. And most important, because the app now introduces new risks that were not addressed in the current SPS or ACP, the vendor might have to obtain a new premarket submission or application and submit an updated SPS and ACP.

The agency hopes that by outlining these two scenarios — one in which the company requires additional FDA clearance or approval and one in which it does not &mdash; it will convince developers that it’s becoming more flexible and is fostering innovation. The proposed framework was released in April 2019 for public comments; the comment period closed on June 3. 

medical devices, not just software that acts as a medical device. Others asked for better definitions of machine learning and more clarity on terms like “locked down” and “continuous learning” algorithms. Many wanted the agency to provide more examples of how changes in AI/ML software might trigger an expectation of premarket review. Allison Fulton, a lawyer with the Life Sciences and FDA team at SheppardMullin, believes that, “While the industry largely encourages FDA to react quickly to evolving technology so as not to stifle innovation, the agency will have to take time to more clearly define foundational terms in this complex area and clarify its expectations for software developers and medical device companies.”

It’s impossible to predict how the FDA will respond to all these comments. And the departure of Scott Gottlieb, M.D., as commissioner adds to this uncertainty. But it seems the policies and philosophy put in place by Gottlieb may well continue to steer the ship. According to 

Aaron Josephson, who spent a decade at FDA

, “… I do not expect the tone and agenda to change with whoever is named acting commissioner or is ultimately confirmed to replace Dr. Gottlieb. [Ned Sharpless, M.D., is now acting Commissioner.] During my time at the FDA, I worked with the team of staff who helped Dr. Robert Califf transition into his role as commissioner. Similarly, I was on the team of staff who aided the Trump administration's early FDA appointees (before Dr. Gottlieb was confirmed) in understanding the agency and prepared Dr. Gottlieb for his return to the agency. During both transition periods, the agency's core business — reviewing applications for new medical products and protecting marketed medical products and food &mdash; continued without issue thanks to the efforts of the FDA’s talented career staff and the statutory mandates FDA is required to satisfy.”

One reason to be optimistic about the FDA’s position on SaMD is the fact that much of SaMD’s future has already been decided by Congress with the passage of the 21st Century Cures Act, passed in December 2017 to encourage innovation. Section 3060, which deals specifically with regulation of medical devices, “expressly excluded from the definition of medical device five distinct categories of 

,” according to one expert. After the Cures Act was passed, the FDA released several guidelines to help developers and clinicians understand its implications, including a set of guidelines on SaMD.

Attorneys with Arnold & Porter explain that the guidelines established a three-step clinical evaluation process that “emphasizes that the level of evaluation and independent review should be commensurate with the risk posed and encourages manufacturers to use continuous monitoring to understand and modify software based on real-world performance.” The key word here is risk. While entrepreneurs planning to invest in SaMD would be wise to seek the advice of lawyers with expertise in the intricacies of FDA regulations, the agency is making it clear that it wants to give developers more room for innovation, especially if their products pose little risk of harming patients.

Replacing Old-School Algorithms with New-School AI in Medicine

Paul Cerrato, M.A., and John Halamka, M.D., M.S., analyze the future of software as a medical device.

How Will New FDA Guidelines Affect Digital Health?

The toll that depression, anxiety, psychosis and related disorders have taken on society is immeasurable. One can, of course, quote 

: About 46 million U.S. adults experience a mental disorder in any given year; 18% of adults experience an anxiety disorder; and 7% have had at least one major depressive episode in the past year.

While these figures are disturbing, the full impact of mental disease really strikes an emotional chord when you know friends, family or business associates who battle these problems. These disorders wreck lives by destroying promising careers, ending marriages, threatening domestic abuse and much more. Unfortunately, with so many individuals suffering from psychiatric problems, it is virtually impossible for every affected patient to receive the one-on-one professional care they need. There just aren’t enough mental health professionals available.

With that in mind, many patients are turning to online and mobile health (mHealth) resources for help. Similarly, several psychiatric experts are taking a closer look at these digital tools to evaluate their potential value, as well as any possible adverse effects.

A Closer Look at Mental Health App Evidence

The evidence on mental health apps and chatbots is mixed. 

, Ph.D., with the Department of Psychiatry and Behavioral Sciences at the Stanford School of Medicine, and her associates have studied 

, a conversational, text-based agent that uses the principles of cognitive behavioral therapy (CBT) to assist users experiencing depressive symptoms. They performed a randomized controlled trial (RCT) to compare the chatbot to a control group who had access to an ebook on mental health.

They found that the program delivered measurable improvements on the nine-item Patient Health Questionnaire, the seven-item Generalized Anxiety Disorder scale and the Positive and Negative Affect Scale among college students. It is important to note, however, that these students had not been officially diagnosed with depression or anxiety.

 uses natural language processing to create a conversational “counselor” that makes users comfortable revealing some of their most personal thoughts. Many patients with depressive symptoms apparently like this approach because they are too embarrassed discussing these issues with a real human being. It is estimated that as many as three out of four college students do not get the medical services they need, and it is not because the services are unavailable or expensive. 

 approximately 70% of teenagers are interested in mobile apps to help them monitor and manage psychiatric problems.

 on mental health chatbots evaluated 10 studies and concluded: “Overall, potential for conversational agents in psychiatric use was reported to be high across all studies. In particular, conversational agents showed potential for benefit in psychoeducation and self-adherence. In addition, satisfaction rating of chatbots was high across all studies, suggesting that they would be an effective and enjoyable tool in psychiatric treatment.” But on a more cautionary note, the analysis also found that, “given the heterogeneity of the reviewed studies, further research with standardized outcomes reporting is required to more thoroughly examine the effectiveness of conversational agents.”

One of the challenges for clinicians, patients, and consumers seeking mobile apps that address mental health is that many rely on unproven therapy techniques. Woebot is based on the principles of CBT, one of the few evidence-based approaches to mental health that has actually generated verifiable results. However, a 

 that are intended for patients with clinical depression is less than encouraging. It reviewed 117 programs that focused on depression and found only 12 of them (10.3%) offered CBT or behavioral activation, another evidence-based treatment approach. The analysis found that those apps that focused on CBT concentrated on education about depression; an explanation of how the CBT approach works; depression rating; monitoring cognitions; emotions, physical sensations and behaviors; conceptualization; and behavioral and cognitive techniques. The core ingredients of behavioral activation also include education about depression and explanation of the model, as well as “depression rating, activity monitoring, giving each activity a rating for pleasure, giving each activity a rating for mastery, activity scheduling of pleasant behaviors and activity scheduling of avoided behaviors.”

Among the many mobile apps targeting the mental health community, Deprexis stands out. It too uses CBT, as well as other well-documented approaches to mental health. It generates simulated conversations that last up to an hour.

 found that this digital “therapist” was moderately effective in relieving depressive symptoms. The evidence supporting Deprexis was strong enough to prompt Great Britain’s relatively 

conservative National Institute for Health and Care Excellence

 (NICE) to recommend that the National Health Service conduct an in-depth evaluation of the app and online program because it may serve as an “effective alternative therapy for adults with mild to moderate depression.”

 performed by Mary Rodgers, Ph.D., M.S. and her colleagues at the University of Michigan’s Department of Internal Medicine found that for every four depressive persons using Deprexis, one recovered, when compared to those who didn’t use the program. Critics still believe these findings should be duplicated in real-world clinical settings, as opposed to those found in a research protocol.

While Deprexis has attracted much attention among mental health professionals, it’s by no means the only mobile app that holds promise. 

Joseph Firth, Ph.D., an Australian investigator

 with Western Sydney University, Campbelltown, and his associates analyzed 18 RCTs and 22 mobile mental health apps and found overall, the apps significantly reduced depressive symptoms with a moderate positive effect size when compared to inactive control participants. An inactive participant refers to someone who didn’t receive any treatment during a given study, compared to an active control, who either used an app that was not designed to treat depression, was treated in person or was engaged in other activities. Firth and his colleagues found less of a difference between patients using a mental health app and someone in an active control group.

A closer look at this meta-analysis, including several subgroup analyses, reveals important insights. The RCTs that looked at smartphone apps lasted from four to 24 weeks, and depressive symptoms were measured using a variety of well-documented tools, including the Depression Anxiety Stress Scale, the Patient Health Questionnaire (PHQ-9) and the Beck Depression Inventory II scale. The meta-analysis also found that the mobile apps were only effective in users who had self-reported mild-to-moderate depression. They had no significant impact on patients with major depression, bipolar disorder or anxiety disorders.

Some of the subgroup analyses were unexpected. Apps that did not involve any in-person feedback generated statistically significant moderately positive effects, while those that did include human feedback did not. Apps that delivered their content entirely through a mobile device appear to have been more effective than those that were not self-contained, though the difference was just short of reaching statistical significance. Finally, those apps that offered cognitive training had less of an impact on users than those that focused more generally on

, conducted by Jesse Wright, M.D., Ph.D., with the University of Louisville School of Medicine in Kentucky, and his associates specifically evaluated computer-assisted CBT. They looked at 40 RCTs and found a moderately large treatment effect. They also noted that apps in which patients were supported by a clinician or other professional were more effective than apps that were used by patients without any outside help.

 that looked at the value of mental health apps, performed by German investigators, included 12 studies; it too found that the programs improved depressive symptoms. It’s important to point out, however, that none of the clinical trials that were evaluated lasted for more than 12 weeks, so it’s not possible to know whether these apps have any long-term impact. One unique aspect of this meta-analysis is that it also looked at health professionals’ attitudes toward mental health apps. The survey of 72 clinicians found that: “Significant differences were found between the level of technology experience and how much the healthcare professional would consider the use of mobile applications in clinical practice. Survey participants reported openness towards therapeutic app use but very little knowledge and experience in the field.”

Although RCTs are considered the gold standard to determine the effectiveness of any medical treatment, that doesn’t mean we should disregard less rigorous types of evidence. A mobile app called Mobilyze, for instance, which was developed by the Center for Behavioral Intervention Technologies at Northwestern Medicine, may have merit according to a 

 in which seven patients with major depression saw significant improvements in depressive and anxiety symptoms at the end of eight weeks. Also encouraging was the fact that the patients no longer met the criteria for depression spelled out in the Patient Health Questionnaire (PHQ-9).

Don’t Ignore the Therapeutic Relationship

There may be significant research support for mental health apps, but clinicians and the public need to keep in mind that a mobile app can never replace the therapeutic relationship that develops between a mental health professional and their patients. While standalone apps may benefit the casual user experiencing mild depressive symptoms, it is unlikely it will be a substitute for the kind of trusting, respective alliance that occurs in a doctor/patient relationship.

, an international authority on digital psychiatry and director of the digital psychiatry division, in the Department of Psychiatry at Beth Israel Deaconess Medical Center in Boston, points out: “The digital therapeutic relationship for mobile health has been ignored because it is often invisible to those building apps. App developers create impressive apps that are marketed to clinicians or patients as a discrete tool….While there isn’t a consensus definition of the therapeutic relationship, elements include mutual trust, alliance, respect, empathy and positive regard between the patient and clinician. The current therapeutic relationship is developed in the brick-and-mortar clinic and strengthened through future in-person clinical visits. The nature of the clinical workflow, appointments, electronic medical records, liability and billing reinforces the predominance of face-to-face therapeutic relationships.”

Torous emphasizes the importance of reframing mental health apps so that they can serve as complementary tools to strengthen the bond between therapist and patient rather than destroy it by encouraging patients to go it alone with an independent computer program — no matter how lifelike and empathetic its conversational interface may seem.

Professional Criteria Can Help Evaluate Apps

Clinicians in need of assistance as they sift through the many mental health apps can turn to the American Psychiatric Association for a 

. APA has published a list of five criteria to help users choose an app that fits their needs and those of their patient population.

While APA does not recommend specific phone apps, its five-step evaluation process can help narrow down one’s choices. The association recommends first collecting background information and then assessing the app’s risks, including the program’s potential to compromise patient privacy. Then review the evidence on effectiveness, evaluate its ease of use and finally evaluate its clinical integration — including interoperability &mdash; and how well it strengthens the therapist/patient relationship.

APA explains, “the last step in the model is interoperability. This is the topmost level, as the ability to share data only matters if this is an app that you and the patient want to use… if it is safe and secure… has some evidence base … and is easy to use…. The reason why interoperability becomes important in this model is because apps should not fragment care and the patient and psychiatrist should be able to share and discuss data or feedback from the app as appropriate. In some cases, the ability for apps to share data may not be relevant. For other apps, however (e.g., mood trackers and medication management), ensuring that such data can be easily shared and accessed by those who need to see it is an important factor to consider.”

Paul Cerrato has more than 30 years of experience working in healthcare as a clinician, educator and medical editor. He has written extensively on clinical medicine, electronic health records, protected health information security, practice management and clinical decision support. He has served as editor of Information Week Healthcare, executive editor of Contemporary OB/GYN, senior editor of RN Magazine, and contributing writer/editor for the Yale University School of Medicine, the American Academy of Pediatrics, Information Week, Medscape, Healthcare Finance News, IMedicalapps.com and Medpage Today. The Healthcare Information and Management Systems Society (HIMSS) has listed Mr. Cerrato as one of the most influential columnists in healthcare IT.

John D. Halamka, M.D., leads innovation for Beth Israel Lahey Health. Previously, he served for over 20 years as the chief information officer (CIO) at the Beth Israel Deaconess Healthcare System. He is chairman of the New England Healthcare Exchange Network (NEHEN) and a practicing emergency physician. He is also the International Healthcare Innovation professor at Harvard Medical School. As a Harvard professor, he has served the George W. Bush administration, the Obama administration and national governments throughout the world, planning their healthcare IT strategies. In his role at BIDMC, Dr. Halamka was responsible for all clinical, financial, administrative and academic information technology, serving 3,000 doctors, 12,000 employees and 1 million patients.

AbleTo Is Changing How We Use Behavioral Health Data

Finding mHealth Apps that Doctors Can Trust

An analysis of the evidence for these increasingly popular mHealth apps.

5% of U.S. adult outpatients experience a diagnostic error each year. 

Diagnostic mistakes contribute to one in 10 deaths.

About 12 million adults suffer a diagnostic mishap annually.

These errors fall into four broad categories: Missed diagnosis, misdiagnosis (coming to the wrong conclusion on what causes a patient’s symptoms), delayed diagnosis and overdiagnosis. The last problem has become more prevalent in developed countries because they have the ability to perform an endless array of tests that sometimes detect innocuous variants that don’t signal the presence of disease.

Although these errors cause a great deal of human suffering, reducing their toll is more complicated than it might appear.  The first issue that needs to be tackled is figuring out how to measure the problem. There is no universally accepted metric to determine scope of the diagnostic error epidemic. Some researchers have used medical records reviews, while others rely on malpractice claims data, health insurance data, physician surveys or patient questionnaires. The flaws with all these yardsticks is they take a great deal of time to collect, and since time is money, healthcare providers are looking for a more cost-effective way to measure the incidence of diagnostic errors. Without such a cost-effective measure, it is almost impossible to gauge the effectiveness of potential solutions.

artificial intelligence (AI)-enhanced tools

 to track the diagnostic error dilemma and an awareness of the cognitive mistakes that contribute to it, clinicians can hope to get a handle on the problem.

One approach that is garnering attention is called the SPADE approach, which attempts to measure diagnostic mistakes by coupling individual symptoms with specific diseases. SPADE, which stands for 

Symptoms-Disease Pair Analysis of Diagnostic Error

, uses electronic health records (EHRs) and billing and insurance claims data to measure the rate at which seemingly benign diagnoses precede more serious life-threatening diseases that were overlooked during the initial medical visit.

One such symptom-disease pair that has been linked to misdiagnosis is dizziness and stroke. A patient may come into the emergency department complaining of dizziness, which the physician diagnoses as otitis media, only to have the patient return to the hospital a few days later with a full-blown cerebrovascular accident. Because researchers have established a clear link between the symptom and the disease, hospitals can track the number of times the coupling occurs to estimate how often their practitioners make the mistake. Then hospitals can take the necessary educational and administrative steps to help correct the problem.

Other pairs that are used in this way are headache and aneurysm, chest pain and myocardial infarction, and fainting and pulmonary embolism.

For the SPADE model to work, providers need a large data set of patient information that includes the symptom and disease occurrences. Healthcare organizations also need to capture these data points regardless of where the patients show up after the more serious event happens. If a significant number of patients with the initial benign diagnosis return to a different health system when they experience the more serious outcome disease, that would skew the results. With that in mind, SPADE is most likely to work when a provider has a very wide reach — for example, when it offers clinical care 

 insures its patient population. That ensures that the organization will have the follow-up administrative data needed to link symptoms and the misdiagnosed disorders that follow. The SPADE metric would also work if patient data were pulled from a regional health information exchange.

Of course, measuring the incidence of diagnostic errors is only the beginning. The next step is understanding their many causes. One National Academy of Medicine report, 

contains a long list of contributing issues, including:

Clinicians who ignore patient input regarding signs and symptoms

Inadequate knowledge base among clinicians

Incorrect interpretation of medical information (.e.g., cognitive errors and biases)

Failure to integrate collected medical information into a plausible diagnostic hypothesis (also caused by cognitive errors and biases)

Not properly communicating the diagnosis to patients

Communication problems between testing facilities and clinicians

Poorly designed clinical documentation systems, including EHRs

Inadequate interoperability between providers

Failure to integrate the diagnostic process into clinicians’ normal workflow

Inadequate teamwork among healthcare professionals

While the list may seem overwhelming to anyone trying to reduce the heavy toll taken by diagnostic errors, most fall into two broad categories: Systemwide issues and cognitive issues. Among the systemwide issues that urgently need attention is the poor communication that often exists between providers and patients. Addressing the problem doesn’t necessarily require the latest AI tools, but it does call for more human intelligence — and compassion. It might at first seem counterintuitive to suggest that patients can play a role in the diagnostic process. Many clinicians believe they should play a silent role. But clinicians can learn a great deal from patients, if they are willing to listen and not constantly interrupt them before they tell their entire story. About 

 can be correctly made based on a patient’s history, which is why 

 still makes sense today: “Listen to your patient, he is telling you the diagnosis.”

Poor communication between testing facilities and clinicians, another barrier to diagnostic accuracy, can be remedied with relatively low-tech solutions. Setting up an electronic system that verifies receipt of important lab results doesn’t require the latest machine-learning algorithms. Similarly, administrative staff and allied health professionals can monitor the back and forth between testing labs and physicians.

Addressing Clinicians’ Diagnostic Thinking Errors

The second broad category responsible for diagnostic mistakes — cognitive biases and errors &mdash; can distort an individual clinician’s reasoning process and lead them down the wrong path. The list of potential problems is long and includes anchoring, affective bias, availability bias and premature closure. During anchoring, a diagnostician will fixate on initial findings and stay anchored to this line of reasoning even when contrary evidence suggests it’s best to change direction. The culture of modern medicine gravitates toward this mindset because it encourages physician overconfidence in their own skill set &mdash; and because many physicians, like other leaders, believe the appearance of certainty is the best course of action. Clinicians who are swayed by their positive and negative emotional reactions to patients, on the other hand, are guilty of an affective bias. Availability bias is common among clinicians who see the same disorder over and over within a short time frame or who have done research on a specific disorder, while premature closure occurs when a practitioner is too quick to accept the first plausible explanation for all the presenting signs and symptoms.

One way to avoid these cognitive errors is for clinicians to take a more introspective approach to the diagnostic process, which is sometimes referred to as metacognition or “thinking about thinking.” It involves forcing oneself to step back and take a dispassionate look at the steps one takes during the reasoning process. Psychologists who have studied diagnosticians’ thinking patterns believe that most physicians rely on one of two approaches: Type 1, which is initiative, automatic and stereotypic thinking, and Type 2, which focuses on slow, logical, effortful, calculating reasoning. The dual system came to the public’s attention when Daniel Kahneman won the Nobel Prize in economics for his seminal work on the topic and published 

The Type 1 system is “fast, frugal, requires little effort, and frequently gets the right answer. But occasionally it fails, sometimes catastrophically. Predictably, it misses the patient who presents atypically, or when the pattern is mistaken for something else,” according to 

, an expert in this field. Type 2 thinking is more reflective and takes the time to reason through several possible forks in the road. Experienced clinicians usually rely on a combination of the two approaches and have learned how to switch between fast and slow thinking.

Of course, even the most gifted diagnostician will make mistakes or become overwhelmed with the sheer volume of data in their workflows. As 

, M.D., and Thomas Lee, M.D., point out, “The complexity of medicine now exceeds the capacity of the human mind.”

But AI and machine learning are successfully addressing this issue.

, we discussed several innovative AI solutions to the diagnostic error dilemma, including the use of neural networks and image analysis to help facilitate the diagnosis of melanoma and other types of machine learning algorithms to assist in the diagnosis of colorectal cancer. But these initiatives only scratch the surface.

, for example, have shown that deep-learning algorithms that use machine learning are as effective as trained pathologists in detecting the spread of breast cancer to nearby lymph nodes. They reached that conclusion by training the algorithms on glass slides of healthy and diseased tissue samples that were definitively diagnosed using immunohistohemical staining. A second set of pathology slides was evaluated by the algorithms and 11 pathologists who were under time constraints to complete the diagnosis, to simulate a real-world setting. Another pathologist arrived at a diagnosis without time constraints. Babak Ehteshami Bejnordi and his colleagues found that “the top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC [without time constraints]” (72.4% vs 64.3-80.4%). Some of the algorithms tested were also as effective as the panel of 11 pathologists who had only two hours to make a diagnosis.

 of the Center for Biomedical Informatics Research at Stanford University, and his colleagues have created a machine-learning-based classifier using a data analytics approach called random forest modeling. It tackles a thorny problem that has plagued physicians for years: How does one spot patients with elevated cholesterol who are at very high risk of heart disease due to a genetic defect called familial hypercholesterolemia (FH)? Having this autosomal dominant condition increases the odds of developing atherosclerotic cardiovascular disease twentyfold when compared to individuals with normal LDL cholesterol levels. While FH affects about one in 250 persons, current screening protocols miss more than 95% of patients with the disorder.

The machine-learning classifier detected 84% of patients at the highest probability threshold of having FH among patients who had been cared for at Stanford Health Care. The investigators used structured and unstructured patient data from Stanford’s EHR system to create their classification system and verified its accuracy by applying it to a different patient population, using it to flag FH patients in Geisinger Health System. That’s an important distinction to point out, as many AI projects have fallen short because they have been shown to be reliable only in a narrowly defined 

A consequence of diagnostic errors is that they often land patients back in the hospital after they have been discharged. The SPADE tool we discussed earlier is designed to help reduce that eventuality. Other researchers have studied the best way to prevent avoidable readmissions by developing more accurate statistics on who is most likely to be readmitted. Several traditional risk scores exist to help make this prediction, including LACE, HOSPITAL and Maxim/Right Care scores, none of which are especially reliable. Researchers from the University of Maryland have developed a risk score derived from EHR data, using machine learning that relies on convolutional neural networks and gradient boosting repression. The new algorithm, called the Baltimore or B score, was compared to the more traditional risk scores in three different hospitals. Each hospital was evaluated with a different version of the B score because the tool was calculated with patient data from each institution.

 used area under the receiver operating characteristic curve (AUROC). 

states: “In a ROC curve, the true positive rate (sensitivity) is plotted in function of the false positive rate (100 - specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. The area under the ROC curve (AUC) is a measure of how well a parameter can distinguish between two diagnostic groups (diseased/normal).”

Morgan and associates found that the B score was significantly better than all the traditional scoring systems. For example, at 48 hours after admission, the AUROC for the B score as 0.72, versus 0.63, 0.64 and 0.66 for HOSPITAL, Maxim/Right Care and modified LACE scores respectively for one hospital. Put another way, “the B score was able to identify the same number of readmitted patients while flagging 25.5% to 54.9% fewer patients.”

And that would save a hospital the resources and manpower needed to deliver specialized care to patients who probably didn’t need it because they were unlikely to be readmitted.

Get the best insights in healthcare analytics 

AI Struggles Across Health Systems, Needs Wide-Range Testing

Setting the Stage for Next-Generation mHealth

Two health-tech leaders examine potentially groundbreaking human and computing efforts to stop diagnostic errors.