Abstract
The European Committee on Antimicrobial Susceptibility Testing (EUCAST) recommends two steps for detecting beta-lactamases in Gram-negative bacteria. Screening for potential extended-spectrum beta-lactamase (ESBL), plasmid-mediated AmpC beta-lactamase, or carbapenemase production is confirmed. We aimed to validate generative pre-trained transformer (GPT)-4 and GPT-agent for pre-classification of disk diffusion to indicate potential beta-lactamases. We assigned 225 Gram-negative isolates based on phenotypic resistances against beta-lactam antibiotics and additional tests to one or more resistance mechanisms as follows: "none," "ESBL," "AmpC," or "carbapenemase." Next, we customized a GPT-agent with EUCAST guidelines and breakpoint table (v13.1). We compared routine diagnostics (reference) to those of (i) EUCAST-GPT-expert, (ii) microbiologists, and (iii) non-customized GPT-4. We determined sensitivities and specificities to flag suspect resistances. Three microbiologists showed concordance in 814/862 (94.4%) phenotypic categories and were used in median eight words (interquartile range [IQR] 4-11) for reasoning. Median sensitivity/specificity for ESBL, AmpC, and carbapenemase were 98%/99.1%, 96.8%/97.1%, and 95.5%/98.5%, respectively. Three prompts of EUCAST-GPT-expert showed concordance in 706/862 (81.9%) categories but were used in median 158 words (IQR 140-174) for reasoning. Sensitivity/specificity for ESBL, AmpC, and carbapenemase prediction were 95.4%/69.23%, 96.9%/86.3%, and 100%/98.8%, respectively. Non-customized GPT-4 could interpret 169/862 (19.6%) categories, and 137/169 (81.1%) agreed with routine diagnostics. Non-customized GPT-4 was used in median 85 words (IQR 72-105) for reasoning. Microbiologists showed higher concordance and shorter argumentations compared to GPT-agents. Humans showed higher specificities compared to GPT-agents. GPT-agent's unspecific flagging of ESBL and AmpC potentially results in additional testing, diagnostic delays, and higher costs. GPT-4 is not approved by regulatory bodies, but validation of large language models is needed. IMPORTANCE: The study titled "GPT-4-based AI agents-the new expert system for detection of antimicrobial resistance mechanisms?" is critically important as it explores the integration of advanced artificial intelligence (AI) technologies, like generative pre-trained transformer (GPT)-4, into the field of laboratory medicine, specifically in the diagnostics of antimicrobial resistance (AMR). With the growing challenge of AMR, there is a pressing need for innovative solutions that can enhance diagnostic accuracy and efficiency. This research assesses the capability of AI to support the existing two-step confirmatory process recommended by the European Committee on Antimicrobial Susceptibility Testing for detecting beta-lactamases in Gram-negative bacteria. By potentially speeding up and improving the precision of initial screenings, AI could reduce the time to appropriate treatment interventions. Furthermore, this study is vital for validating the reliability and safety of AI tools in clinical settings, ensuring they meet stringent regulatory standards before they can be broadly implemented. This could herald a significant shift in how laboratory diagnostics are performed, ultimately leading to better patient outcomes.