Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non-English Languages
Recommended Citation
Oh E, Wu AW, Garcia-Rodriguez L, Huang ZX, Hsu WC, Kim J, Kim YC, Lin YT, Park HC, Perez HA, Juan JS, Liu G, Lee M, Tang DM. Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non-English Languages. World J Otorhinolaryngol Head Neck Surg. 2026;12(5):553-558.
Document Type
Article
Publication Date
10-1-2026
Publication Title
World J Otorhinolaryngol Head Neck Surg
Abstract
BACKGROUND: Large language models (LLMs) such as ChatGPT are being explored for various medical applications, but their performance across languages is not established. Despite the promising results in English from previous studies, it is unclear how ChatGPT performs in other languages. We assessed the ability of ChatGPT-4 to answer common otolaryngology-related frequently asked questions in Chinese, Korean, and Spanish in this study.
METHODS: Eighteen clinical questions were selected from the American Academy of Otolaryngology-Head and Neck Surgery Foundation guidelines and ENT Health across six subspecialties. Bilingual clinicians translated the questions to Chinese, Korean, or Spanish and input them into ChatGPT-4 (OpenAI, San Francisco, USA). The responses were rated on a five-point Likert scale for accuracy, completeness, and similarity to English reference answers. A linear mixed-effects model with language as a fixed effect and random intercepts for question and rater was used to compare performance across languages. Inter-rater reliability was assessed using intraclass correlation coefficients and Krippendorff's
RESULTS: ChatGPT-4's performance differed significantly by language in all assessment domains. Responses in Spanish scored highest throughout and most frequently were characterized as clear and organized. Korean answers were generally accurate but tended to be more conservative and less detailed. Chinese answers showed greater variability, with reviewers noting a mix of clear and less accurate outputs.
CONCLUSION: The clinical performance of ChatGPT-4 differed across Chinese, Korean, and Spanish. These findings underscore the importance of ongoing evaluation and refinement of LLMs to ensure they provide reliable multilingual support in clinical practice.
PubMed ID
42609711
ePublication
ePub ahead of print
Volume
12
Issue
5
First Page
553
Last Page
558
