Assessing ChatGPT’s Viability as a Diabetes Consultant
In a recent investigation featured in the journal PLoS ONE, researchers delved into the potential and challenges of utilizing ChatGPT, a conversational AI model, as a diabetes consultant to address common inquiries related to this chronic condition.
The burgeoning field of artificial intelligence, with ChatGPT at its forefront, has piqued considerable interest for its potential in clinical applications. While not initially designed for healthcare, ChatGPT boasts millions of active users worldwide. Studies have revealed that individuals are increasingly open to AI-driven solutions for low-risk situations, showcasing a growing acceptance of these technologies. This has prompted a deeper exploration into the utility and usage of expansive language models like ChatGPT in everyday clinical contexts.
Tip: Please fill out this form to determine whether or not you or a friend are eligible for a CGM.
Study Overview
In this study, researchers aimed to assess ChatGPT’s proficiency in responding to diabetes-related queries, particularly those frequently posed by patients, in a manner comparable to human experts.
The research specifically investigated whether individuals with varying levels of expertise in diabetes could differentiate between responses generated by humans and those crafted by ChatGPT. Additionally, the study explored whether participants with prior experience as healthcare providers for diabetes patients or individuals who had previously interacted with ChatGPT were more adept at identifying responses generated by the AI model.
The research employed a closed Turing test-inspired online survey, distributed to all Steno Diabetes Center Aarhus (SDCA) staff, whether part-time or full-time employees. The survey comprised ten multiple-choice questions, each with two possible answers—one composed of humans and the other generated by ChatGPT. Demographic data, including age, gender, and prior interaction with ChatGPT users, were also collected. Participants were tasked with discerning the ChatGPT-generated responses.
The ten questions spanned topics such as the underlying mechanisms of diabetes, treatment options, potential complications, physical activity, and dietary considerations. Eight of these questions were drawn from the ‘Frequently Asked Questions’ section of the Diabetes Association of Denmark’s website as of January 10, 2023. The remaining questions were designed to align with specific content from the ‘Knowledge Center for Diabetes website and a report on physical activity and type 1 diabetes mellitus.
Read Guide about Wegovy Dosage Guide: The Best Way For Weight Loss
Statistical analysis was conducted using logistic regression modeling to determine odds ratios (ORs). The research team examined the impact of various participant characteristics on the study’s outcomes. A non-inferiority margin of 55%, defined and disclosed in the research protocol before data collection, guided the analysis. Human-authored responses were directly sourced from relevant materials or reference websites containing the questions. More About Our Top CGMs
To maintain consistency in response length, two healthcare experts trimmed some answers to match the desired word count. Prior to presenting the questions, context along with three random samples from the 13 pairs of questions and answers were provided to the AI-based language model in the prompts. Participants received personalized survey links via email, and data collection occurred from January 23 to 27, 2023.
Must Read About Oral Semaglutide: A Potential Solution for Obesity in Mice
Study Results
Among the 311 individuals invited to participate, 183 completed the survey, yielding a response rate of 59%. Of these respondents, 70% were female, 64% were aware of ChatGPT, 19% had previous experience using it, and 58% had interacted with diabetes patients in their healthcare roles. While the AI model was instructed to provide responses containing 45 to 65 words to match human answers, the average response length was 70 words. To ensure fairness, consultation recommendations and the initial three lines of questions were omitted from the ChatGPT-generated responses, resulting in an average response length of 56 words.
Accuracy in identifying ChatGPT-generated responses across the ten questions varied from 38% to 74%. Participants correctly recognized ChatGPT-generated answers 60% of the time, surpassing the predefined non-inferiority threshold. Specifically, males and females achieved recognition rates of 64% and 58%, respectively. Participants with prior experience interacting with diabetes patients demonstrated a 61% recognition rate, compared to 57% for those lacking such experience.
The strongest association with correct recognition of ChatGPT-generated responses was found among participants who had previously used ChatGPT (OR, 1.5). A similar odds ratio was observed for participants over 50 years of age, who were more likely to accurately identify AI-generated responses (OR, 1.3). Previous ChatGPT users correctly answered 67% of the questions, whereas non-users achieved a 58% accuracy rate. Contrary to initial assumptions, participants demonstrated an ability to distinguish between responses generated by ChatGPT and those crafted by humans, outperforming a random guessing scenario.
Conclusion
In summary, this study represents an initial exploration into the capabilities and constraints of ChatGPT as a provider of patient-centric guidance, particularly in managing chronic diseases like diabetes. While GPT exhibited promise in accurately addressing frequently asked questions, issues related to misinformation and the absence of nuanced, personalized advice were apparent. As large language models increasingly intersect with healthcare, rigorous research is imperative to evaluate their safety, effectiveness, and ethical implications in patient care. This underscores the necessity for robust regulatory frameworks and continuous oversight.
