From The Stacks
EDITOR’S NOTE: There are literally thousands of journals published around the world that relate to the disability community. It is virtually impossible to capture even a fraction of them. HELEN receives "stacks" of journals and selectively earmarks what we feel are "must read" articles of interest for our readers. It's a HELEN perk!
NIH Findings Shed Light on Risks and Benefits of Integrating AI Into Medical Decision-making
GPT-4V, an AI model, often made mistakes when describing the medical image and explaining its reasoning behind the diagnosis—even in cases where it made the correct final choice.NIH/NLM
Researchers at the National Institutes of Health (NIH) found that an artificial intelligence (AI) model solved medical quiz questions—designed to test health professionals’ ability to diagnose patients based on clinical images and a brief text summary—with high accuracy. However, physician-graders found the AI model made mistakes when describing images and explaining how its decision-making led to the correct answer. The findings, which shed light on AI’s potential in the clinical setting, were published in npj Digital Medicine(link is external). The study was led by researchers from NIH’s National Library of Medicine (NLM) and Weill Cornell Medicine, New York City.
“Integration of AI into health care holds great promise as a tool to help medical professionals diagnose patients faster, allowing them to start treatment sooner,” said NLM Acting Director, Stephen Sherry, Ph.D. “However, as this study shows, AI is not advanced enough yet to replace human experience, which is crucial for accurate diagnosis.”
“AI model scored well on medical diagnostic quiz, but made mistakes explaining answers.”
The AI model and human physicians answered questions from the New England Journal of Medicine (NEJM)’s Image Challenge. The challenge is an online quiz that provides real clinical images and a short text description that includes details about the patient’s symptoms and presentation, then asks users to choose the correct diagnosis from multiple-choice answers.
The researchers tasked the AI model to answer 207 image challenge questions and provide a written rationale to justify each answer. The prompt specified that the rationale should include a description of the image, a summary of relevant medical knowledge, and provide step-by-step reasoning for how the model chose the answer.
Nine physicians from various institutions were recruited, each with a different medical specialty, and answered their assigned questions first in a “closed-book” setting, (without referring to any external materials such as online resources) and then in an “open-book” setting (using external resources). The researchers then provided the physicians with the correct answer, along with the AI model’s answer and corresponding rationale. Finally, the physicians were asked to score the AI model’s ability to describe the image, summarize relevant medical knowledge, and provide its step-by-step reasoning.
The researchers found that the AI model and physicians scored highly in selecting the correct diagnosis. Interestingly, the AI model selected the correct diagnosis more often than physicians in closed-book settings, while physicians with open-book tools performed better than the AI model, especially when answering the questions ranked most difficult.
Importantly, based on physician evaluations, the AI model often made mistakes when describing the medical image and explaining its reasoning behind the diagnosis — even in cases where it made the correct final choice. In one example, the AI model was provided with a photo of a patient’s arm with two lesions. A physician would easily recognize that both lesions were caused by the same condition. However, because the lesions were presented at different angles — causing the illusion of different colors and shapes — the AI model failed to recognize that both lesions could be related to the same diagnosis.
The researchers argue that these findings underpin the importance of evaluating multi-modal AI technology further before introducing it into the clinical setting.
“This technology has the potential to help clinicians augment their capabilities with data-driven insights that may lead to improved clinical decision-making,” said NLM Senior Investigator and corresponding author of the study, Zhiyong Lu, Ph.D. “Understanding the risks and limitations of this technology is essential to harnessing its potential in medicine.”
The study used an AI model known as GPT-4V (Generative Pre-trained Transformer 4 with Vision), which is a ‘multimodal AI model’ that can process combinations of multiple types of data, including text and images. The researchers note that while this is a small study, it sheds light on multi-modal AI’s potential to aid physicians’ medical decision-making. More research is needed to understand how such models compare to physicians’ ability to diagnose patients.
The study was co-authored by collaborators from NIH’s National Eye Institute and the NIH Clinical Center; the University of Pittsburgh; UT Southwestern Medical Center, Dallas; New York University Grossman School of Medicine, New York City; Harvard Medical School and Massachusetts General Hospital, Boston; Case Western Reserve University School of Medicine, Cleveland; University of California San Diego, La Jolla; and the University of Arkansas, Little Rock.
The National Library of Medicine (NLM) is a leader in research in biomedical informatics and data science and the world’s largest biomedical library. NLM conducts and supports research in methods for recording, storing, retrieving, preserving, and communicating health information. NLM creates resources and tools that are used billions of times each year by millions of people to access and analyze molecular biology, biotechnology, toxicology, environmental health, and health services information. Additional information is available at https://www.nlm.nih.gov.
About the National Institutes of Health (NIH): NIH, the nation's medical research agency, includes 27 Institutes and Centers and is a component of the U.S. Department of Health and Human Services. NIH is the primary federal agency conducting and supporting basic, clinical, and translational medical research, and is investigating the causes, treatments, and cures for both common and rare diseases. For more information about NIH and its programs, visit www.nih.gov.
Reference
Qiao Jin, et al. Hidden Flaws Behind Expert-Level Accuracy of Multimodal GPT-4 Vision in Medicine. npj Digital Medicine. DOI: 10.1038/s41746-024-01185-7(link is external) (2024).
(From NIH…Turning Discovery Into Health)
Investing in Oral Health: Here’s Why the Time is Now
Oral diseases affect nearly half the world’s population, posing significant risks to individual health and systemic illnesses and contributing to substantial economic costs and productivity losses globally.
There is growing recognition of the importance of oral health, highlighted by the adoption of the World Health Organization’s global strategy on oral health in 2022 and several leading on broader public health integration.
There is a critical need for increased investment from public and private sectors to improve oral health and address affordability, accessibility and advancement.
What would you guess is the most common noncommunicable disease worldwide? If you guessed cancer or cardiovascular disease, you’d be wrong – but you wouldn’t be alone.
Almost half the world’s population – 3.5 billion people – are impacted by oral diseases that not only imperil their ability to eat or speak but also put them at greater risk of and from other systemic illnesses like diabetes, heart disease, stroke and some cancers. The resulting cost to the global economy is staggering, estimated at more than $710 billion for treatment costs plus productivity losses combined.
In comparison, oral health programmes focused on prevention are highly impactful at low cost. Yet, until recently, oral health has largely been neglected by the public and private sectors and the broader global health agenda.
Pivotal change in oral health policy
Following a growing global consensus on the importance of oral health to overall health, a tipping point came in 2022 with the adoption of the World Health Organization’s landmark global strategy on oral health and the release of its global oral health status report, which highlights the cost of this neglect – and the starkly higher burden it imposes on socio-economically disadvantaged and underrepresented populations.
More recently, the FDI World Dental Federation, representing over 1 million dentists worldwide and 191 national dental associations in 134 countries, adopted a new vision calling for integrating oral health into primary care and including dental care in universal health coverage policies.
Yet despite this growing recognition of the importance of oral health, access to care remains tragically constrained by governments’ lack of investment in dental facilities, the dental workforce and ensuring the affordability of needed care, compounded by employers’ lack of investment in employee oral health.
Largely the result of the historic siloing of oral health and medicine, these investment choices have made it impossible to stem the tide of oral disease and keep people healthy.
Mobilizing oral health action
The World Economic Forum Global Health Equity Network’s Oral Health Affinity Group is working to help change this. Composed of leaders across sectors and industries, we are committed to raising awareness of the costs of poor oral health to individuals, communities, businesses and nations.
Of prime importance to this group is the collective exploration of actions that governments and private sector companies can take to advance global oral health.
The Economic Rationale for a Global Commitment to Invest in Oral Health is the Forum’s first publication aimed at catalyzing prompt action by the public and private sectors. It makes plain the enormous costs we pay individually, as a society and globally for poor oral health.
Lost school and work days, lower academic achievement, and diminished job prospects for adults are some impacts – penalties that are most pronounced for low-income, vulnerable populations make oral health a clear health equity priority. Also felt are diminished labour productivity, with losses stemming from dental diseases similar to those for Alzheimer’s disease and other dementias.
On the flip side, better oral health benefits individuals and communities improves productivity, lowers acute healthcare spending and creates healthcare cost savings among populations with concomitant noncommunicable diseases.
Leading on oral health
There are already leaders showcasing how investing in oral health can benefit communities, economies and broader society. For example, Thailand, Japan and Brazil are a vanguard in adopting universal public health insurance schemes and including oral health benefits in that coverage.
As a result, an estimated 70% of Brazilians now access healthcare services yearly with significant improvements in oral health indicators, including dramatic reductions in tooth loss due to dental disease.
In Japan, because dental care service delivery is now also integrated into the broader healthcare delivery model, dentists work cooperatively with other healthcare professionals to improve the care of patients with certain medical conditions such as diabetes and dementia, help identify patients with these diseases and administer appropriate care to prevent disease progression.
In the United States, Medicaid expansion in some states to provide dental care coverage for adults has resulted in reducing patients’ risk of being diagnosed with diabetes and heart disease, which reduced costs to the Medicaid programme. The expansion also reduced both emergency room spending and inpatient healthcare costs.
Investing in oral health
The bottom line: we can’t afford not to invest in oral health. That’s why we call on governments to take a key role in improving oral health, including:
Implementing policies that make oral health care more affordable and accessible, such as integrating oral health within primary health care and public health insurance programmes.
Promoting oral health literacy.
Helping to ensure a robust oral health workforce.
There is also an important opportunity for private-sector employers to help expand access to care by
Incorporating oral care and education into employee benefits.
Spearheading technological advances in oral care.
Investing in affordable products with the potential to improve prevention, especially in under-resourced countries and communities.
Vitally, each sector cannot work in silos and we must join efforts through innovative public-private partnerships to create lasting change. Only if we come together to make the necessary investments in oral health will we achieve substantially improved global access to preventive oral care, a reduced burden of noncommunicable diseases and ultimately, a healthier world.
(From The World Economic Forum website)
