Evaluation Of Medical Diagnosis Capabilities Of Three Artificial Intelligence Models – ChatGPT-3.5, Google Gemini, Microsoft Copilot

Abstract

The widespread adoption of artificial intelligence (AI) in various domains, including medicine,
has prompted extensive research into its diagnostic capabilities. This study conducts a comparative
analysis of three prominent AI models – ChatGPT-3.5, Microsoft Copilot, and Google Gemini – to
evaluate their performance in medical diagnosis. Clinical vignettes from Texas Tech University Health
Sciences Center were utilized to assess the accuracy and precision of the AI models in diagnosing internal
medicine cases. Results indicate that ChatGPT-3.5 achieved the highest accuracy rate, correctly
diagnosing 70.59% of cases, outperforming Google Gemini and Microsoft Copilot. While all models
demonstrated the potential to assist in diagnosis, variations in approach and performance were observed.
ChatGPT-3.5 provided concise answers without explicitly stating its lack of medical expertise, while
Google Gemini and Microsoft Copilot acknowledged their limitations but offered more detailed
explanations and recommendations. Statistical analysis, conducted using the chi-square test for
independence revealed significant differences in diagnostic capabilities among the AI models,
emphasizing the importance of careful selection in clinical decision-making. This study contributes
valuable insights into the application of AI in medical diagnosis and underscores the need for continued
refinement of AI models to enhance diagnostic accuracy and support healthcare professionals in
delivering optimal patient care.