Abstract
BACKGROUND: Research-grade artificial intelligence has been used to accurately diagnose strabismus from image input. OpenAI's consumer-oriented GPT-4o model can analyze images, but has shown poor accuracy for image-based diagnosis. Parents may turn to GPT-4o to support or refute visible health concerns for their children, such as strabismus. The study aims to evaluate GPT-4o's diagnostic accuracy and response quality for strabismus evaluation. METHODS: After gold-standard alternate cover exam by a clinician, 35 mobile photos of esotropia (13), pseudoesotropia (11), and exotropia (11) were selected. Images were excluded if a second "masked" examiner did not corroborate diagnosis. Images were submitted to a secure GPT-4o platform with patient-perspective prompts requesting overall evaluation (Prompt 1) and eye alignment evaluation (Prompt 2). Responses were graded by a pediatric ophthalmologist and certified orthoptist assessing for quality and safety. RESULTS: GPT-4o provided interpretations for 15/35 and 27/35 images after Prompts 1 and 2, respectively. Analysis of the accuracy includes a primary "intention-to-diagnose" and secondary "per-diagnosis" framework. The diagnostic accuracies in the primary and secondary analysis following prompt 1 were 14.3% (low sensitivity and specificity) and 33.3% (low sensitivity, high specificity), respectively. Following prompt 2, accuracies were 48.6% (moderate sensitivity, low specificity) and 63.0% (high sensitivity, low specificity). Overall, the mean rating for content for strabismus prompts was 4.94 ± 0.27 out of a best possible 6, and for pseudostrabismus, 5.14 ± 0.18 (p=0.638). CONCLUSION: GPT-4o shows poor accuracy for image-based strabismus diagnosis. GPT-4o frequently categorized pseudoesotropia as true strabismus and true strabismus as orthophoria. While the quality of responses was rated as good overall, the quality of counseling did not match what would be provided in a pediatric ophthalmology clinic. Patients, clinicians, and AI developers should be aware of the need for specialist evaluation for strabismus.