Abstract
INTRODUCTION: In chronic diseases the length and redundancy of medical records can hamper the accurate interpretation of the individual clinical history, and we hypothesize that Natural Language Processing (NLP) may be used to optimize information from unstructured texts in electronic health records (EHRs). We developed an NLP-based classifier to discriminate rheumatoid (RA) and psoriatic arthritis (PsA) versus non-inflammatory rheumatic diseases (NIRD), namely osteoarthritis and fibromyalgia, from notes in the rheumatology practice. METHODS: A retrospective cross-sectional study was conducted on 1,372 anonymized EHRs from a single Italian rheumatology center, including 468 RA, 732 PsA, and 172 NIRD cases. After removing explicit diagnostic referrals, texts were embedded using five models and the resulting embeddings were used to train multiple classifiers. The best-performing embedding-classifier combination was selected, predictions were stratified into high-, moderate-, and low-certainty according to a confidence-based decision system, and explainability was assessed using LIME and SHAP. RESULTS: The chosen model reached an overall accuracy of 85% in distinguishing between the three diagnostic groups. In detail, 74.5% of cases were classified with high certainty, reaching 94.6% accuracy, while only 1.5% were considered uncertain. No misclassifications occurred between RA and NIRD, though some overlap was seen between PsA and the other two conditions. Attempts to interpret the model’s decisions through dedicated algorithms provided partially meaningful insights, but results were inconsistent, especially when the clinical notes included complex or negated language. CONCLUSIONS: We developed an NLP-based classifier to detect inflammatory arthritis from unstructured rheumatology notes. The model showed high confidence and accuracy, with low risk of missing patients with inflammatory disease. Although interpretability remains a limitation, external validation in multicenter and multilingual cohorts is warranted to confirm its generalizability. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s13075-025-03713-6.