Abstract
INTRODUCTION: Large language models (LLMs) offer remarkable potential in assisting healthcare professionals with diagnostic and therapeutic decision-making processes. However, the integration of LLMs into healthcare decision-making processes also introduces several doubts in the field of usefulness, reliability and ethical implications. MATERIALS AND METHODS: To assess the potential of LLMs in managing intricate surgical situations, a cross-sectional study with 30 real-world cases of maxillofacial traumatology was designed. The cases were presented to ChatGPT-4, Google Bard, and maxillofacial surgery residents in a standardized manner, and the results of the subjects were evaluated by an expert surgeon panel of reviewers using the AIPI and QAMAI tools. RESULTS: ChatGPT-4 and Bard showed comparable performances in patient feature consideration but differed significantly in their ability to suggest differential diagnoses. ChatGPT-4 outperformed Bard in proposing additional examinations and treatment plans. Compared to LLMs, human surgery residents consistently scored higher across all parameters of the QAMAI tool, indicating superior accuracy, clarity, relevance, completeness, quality of references, and overall usefulness. DISCUSSION: Both LLMs demonstrated their potential to support clinical decision-making in facial traumatology, but they require further development to be sufficiently reliable for real-world clinical use. CONCLUSIONS: AIPI and QAMAI proved their utility as evaluation tools but highlighted the need for standardization in LLM-generated responses assessment. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s12663-025-02556-7.