Abstract
Fall-related injuries (FRIs) are a major cause of hospitalizations among older patients, but identifying them in unstructured clinical notes poses challenges for large-scale research. In this study, we developed and evaluated natural language processing (NLP) models to address this issue. We utilized all available clinical notes from the Mass General Brigham health-care system for 2100 older adults, identifying 154 949 paragraphs of interest through automatic scanning for FRI-related keywords. Two clinical experts directly labeled 5000 paragraphs to generate benchmark-standard labels, while 3689 validated patterns were annotated, indirectly labeling 93 157 paragraphs as validated-standard labels. Five NLP models, including vanilla bidirectional encoder representations from transformers (BERT), the robustly optimized BERT approach (RoBERTa), ClinicalBERT, DistilBERT, and support vector machine (SVM), were trained using 2000 benchmark paragraphs and all validated paragraphs. BERT-based models were trained in 3 stages: masked language modeling, general boolean question-answering, and question-answering for FRIs. For validation, 500 benchmark paragraphs were used, and the remaining 2500 were used for testing. Performance metrics (precision, recall, F1 scores, area under the receiver operating characteristic curve [AUROC], and area under the precision-recall [AUPR] curve) were employed by comparison, with RoBERTa showing the best performance. Precision was 0.90 (95% CI, 0.88-0.91), recall was 0.91 (95% CI, 0.90-0.93), the F1 score was 0.91 (95% CI, 0.89-0.92), and the AUROC and AUPR curves were [both??] 0.96 (95% CI, 0.95-0.97). These NLP models accurately identify FRIs from unstructured clinical notes, potentially enhancing clinical-notes-based research efficiency.