Abstract
Background Thyroid nodules are among the most common endocrine abnormalities, with ultrasound serving as the first-line tool for risk stratification. The American College of Radiology Thyroid Imaging Reporting and Data System (ACR-TIRADS) standardizes evaluation but is limited by interobserver variability and the time required for detailed interpretation. Artificial intelligence (AI) offers the opportunity to address these limitations and to automate diagnostic processes and enhance diagnostic accuracy. Objective To develop and evaluate a vision-language AI model for ACR-TIRADS-based risk stratification of thyroid nodules on ultrasound. Methodology This retrospective study analyzed 1,000 thyroid ultrasound images collected between March 2024 and January 2025, of which 139 met the inclusion criteria. Images were annotated according to ACR-TIRADS features (composition, echogenicity, shape, margins, echogenic foci). A vision-language AI model (LLaVA-Med, Microsoft Research, Redmond, WA) was trained using a two-stage strategy that included domain-specific pretraining and fine-tuning on a curated dataset. Diagnostic performance was assessed as a binary classification: suspicious (TR3-TR5) vs. non-suspicious (TR1-TR2). Results The model achieved an accuracy of 67%, sensitivity of 71%, specificity of 53%, and precision of 84.6%. The F1 score is an average of an AI algorithm's precision and recall, used to evaluate the algorithm's predictive performance. Our model achieved an F1 score of 77%, and its performance favored sensitivity, reducing the likelihood of missed malignant nodules, though specificity remained moderate. Conclusion The vision-language AI model trained on ACR-TIRADS features demonstrated promising performance in thyroid nodule risk stratification. Its higher sensitivity and explainable outputs reflect its potential as a supportive screening tool in clinical practice, particularly in settings with limited radiological expertise. Further refinement and multi-institutional validation are warranted.