Abstract
We evaluated the zero-shot performance of six large language models (LLMs; GPT-4.0 Turbo, LLaMA-3-8B, LLaMA-3-70B, Mixtral 8[Formula: see text]7B Instruct, Titan Text G1-Express, Command R+) and four multimodal LLMs (Claude-3.5-Sonnet, Claude-3-opus, Claude-3-Sonnet, Claude-3-Haiku) on the 2023 Brazilian Portuguese medical residency entrance exam of the Hospital das Clínicas da Faculdade de Medicina da Universidade de São Paulo including text-only and image-based questions. Comparison among models showed that accuracy varied widely, with Claude-3.5-Sonnet achieving the highest score on text-only questions (70.27%, 95% CI: 65.68–74.86), surpassing GPT-4.0 Turbo (66.22%, 95% CI: 65.38–67.05), while the open-source LLaMA-3-70B performed competitively. The best models reached the median level observed among human candidates. On image-based questions, accuracy dropped substantially across models, with most scoring below 50%, except Claude-3.5-Sonnet, which maintained stable performance. However, this decline should be interpreted with caution, as it remains unclear whether it reflects multimodal reasoning limitations or differences in intrinsic question difficulty, and the present study does not allow these possibilities to be disentangled. In addition, qualitative analysis by independent expert physicians assessed model-generated explanations, identifying hallucinatory events, with lower inter-rater agreement in misclassified cases. These results suggest that language models in Brazilian Portuguese may approximate human-level reasoning in medical questions.