According to a study, artificial intelligence (AI) chatbots may provide a supplemental option for patient education in oncology, though there is concern about their limitations.

These data were first presented by Sujata Ojha, MD, Dell Medical School of The University of Texas at Austin, Texas, at the 2025 American Society of Clinical Oncology (ASCO) Gastrointestinal Cancers Symposium.

Dr Ojha explained, due to the increase of colon cancer incidence among patients younger than 55 in the United States, and an overall growing reliance on digital technologies for health information, “there is an increasing need to assess whether AI tools like ChatGPT can bridge gaps in patient education and present reliable and accurate responses that align with expert opinion.”

Guided by sources such as the National Cancer Center, the American Cancer Society, and others, the researchers created 10 questions, meant to represent commonly asked questions from patients with colon cancer. The questions were categorized as either “general oncology characteristics,” related to symptoms, screening, and prevention; or “diagnosis and treatment.” Simulating patient inquiries, these questions were entered into ChatGPT, an AI chatbot. The responses generated by ChatGPT were evaluated by oncology experts, using a 5-point Likert scale to reflect the experts’ level of agreement with each response.

On the Likert scale, where 1 represented “strongly disagree” and 5 represented “strongly agree,” the mean score of the responses was 4.72. Across the mean scores from all raters, there was not a statistically significant difference (P = 0.221). However, there was a statistically significant difference between the ratings for the 2 categories (P = .034), with experts agreeing less with the ChatGPT responses related to “diagnosis and treatment” than “general oncology characteristics.” Researchers noted that this difference could be a result of ChatGPT’s reliance on existing information — diagnosis and treatment of colon cancer is rapidly involving and increasing personalized for each specific patient, therefore new and individual information may not be included in the AI algorithm.

Dr Ojha et al, concluded, “ChatGPT can provide accurate and relevant responses to patient inquiries about colon cancer, as assessed by medical oncology experts,” adding that these results, “highlight the potential of AI chatbots in supplementing patient education in oncology, though further research is necessary to explore its limitations and expand its clinical utilities.”

Source:

Ojha S, Sessions W, Goodgame B. Man versus machine: Comparative analysis of ChatGPT’s colon cancer guidance and expert opinion. Presented at 2025 ASCO Gastrointestinal Cancers Symposium. January 23-25, 2025; San Francisco, CA. Abstract 299.

New study examines accuracy and relevancy of ChatGPT for information regarding colon cancer. News Release. ASCO. News Release. American Society of Clinical Oncology. Accessed January 23, 2025. https://society.asco.org/about-asco/press-center/news-releases/new-study-examines-accuracy-and-relevancy-chatgpt-information?cid=DM19308&bid=450425113