While the public acknowledges the benefits of medical imaging, their understanding of the associated risks remains limited(1–3). As a result, patients are turning to the internet for answers. The rise of easily accessible artificial intelligence (AI) has transformed the way the public engages with online information, especially through language models like OpenAI’s ChatGPT.
We recently published a comparative study of ChatGPT and radiology institutional websites in providing radiation protection information(4). To our knowledge, this study represented the first attempt to gauge LLMs' potential in the important topic of radiation protection communication and provided a structured assessment of ChatGPT’s performance on a set of questions in this field. Twelve radiation protection-related questions retrieved from institutional websites were prompted into ChatGPT-3.5. Twelve international experts (4 radiologists, 4 medical physicists, and 4 radiographers) blinded to the source evaluated both ChatGPT-generated and institutional responses for scientific accuracy, public comprehension, and satisfaction using a Likert scale (1=No; 7=Yes). No statistically significant differences were found in terms of overall satisfaction, scientific accuracy, and anticipated patient understandability. However, experts perception of whether AI had generated the response was different between Human and ChatGPT responses (p = 0.02). Raters correctly identified the source with high confidence in 43% (62 of 144) of human responses and 61% (88 of 144) of ChatGPT responses, respectively (p < 0.001).
A critical limitation of experiments involving ChatGPT in medicine, including our own, was the absence of the patient's perspective. As with most studies evaluating ChatGPT's performance, experts are judging patient understandability instead of the patients themselves.
New elements in the literature explore direct patient perception. To start with, a recent study demonstrated that 17 rheumatology patients rated AI-generated responses to patient questions similarly to physician-generated responses in terms of comprehensiveness, readability, and overall preference(5). However, one study indicated a negative shift in perception towards AI technology and unmet needs for disease understanding among urolithiasis patients after receiving an explanatory note from an AI chatbot, particularly among those with lower education levels(6). This underscores the fact that, despite patients using Google for health-related information for years, ChatGPT is a relatively new tool. Consequently, we do not yet fully understand patients' expectations and their current use of ChatGPT. An other study(7) highlighted several issues in patient care: patients often receive unclear instructions and lack specific information, leading to information overload; they also tend to have poor recall of medical consultations, and more questions arise later. While patients can turn to online sources for additional information, they are often skeptical and distrust online forums. Given these challenges, it is worth considering whether a form of LLM could play a role in addressing these issues.
The present study aimed to assess the patient perspective of ChatGPT in providing radioprotection information compared to institutional radiology websites.