Evaluation of the Utility of ChatGPT Responses In Addressing the Diagnosis and Queries Related to Pulmonary Embolism

Vivie Tran, Hoang Ho, Subash Swarna, Andrew Ibrahim, Minnie Tran, Steven Daley, Simon Williams, MD, Mohammad M. Ansari, MD

PAD Center of Excellence, Texas Tech University Health Sciences Center, Lubbock, TX, USA

Background

Despite proven utility of ChatGPT in diverse domains, its effectiveness in addressing inquiries related to PE remains uncertain when multiple parameters are presented. Our investigation aims to evaluate the suitability of ChatGPT's responses to fundamental questions concerning PE, focusing on aspects such as risk factors, prevention, and treatment, including medication information.

Methods

To conduct this study, 15 questions were posed to ChatGPT. The queried topics encompassed risk counseling, preventive measures, and details about treatment modalities. An interdisciplinary panel comprising physicians specializing in interventional cardiology, vascular surgery, and interventional radiology reviewed the responses. The panel categorized the answers as appropriate, inappropriate, or unreliable based on their clinical expertise. Two independent teams performed the review, and the results were combined and analyzed for correctness.

Results

ChatGPT delivered appropriate responses for 11/15 questions (73.3%). However, in 4/15 questions (26.7%), the responses were deemed unreliable. These unreliable responses were characterized by a lack of comprehensive details necessary for fully addressing the questions. Questions deemed unreliable predominantly pertained to areas such as signs/symptoms and diagnostic methods.

Conclusions

This study emphasizes the potential of ChatGPT in furnishing primarily appropriate responses to inquiries regarding PE. The application of AI tools holds promise in augmenting patient understanding and quality of care in the context of PE. Nevertheless, ongoing advancements necessitate further research to comprehensively understand the role of ChatGPT. Since AI depends on initial data input, the future might hold physician teams joining to feed data that can allow them to obtain diverse interpretation.