Abstract
BACKGROUND: To determine the impact of electronic health record (EHR)-discontinuity on the performance of prediction models. METHODS: The study population consisted of patients with a history of cardiovascular (CV) comorbidities identified using US Medicare claims data from 2007 to 2017, linked to EHR from two networks (used as model training and validation set, respectively). We built models predicting one-year risk of mortality, major CV events, and major bleeding events, stratified by high vs. low algorithm-predicted EHR-continuity. The best-performing models for each outcome were chosen among 5 commonly used machine-learning models. We compared model performance by Area under the ROC curve (AUROC) and Area under the precision-recall curve (AUPRC). RESULTS: Based on 180,950 in the training and 103,061 in the validation set, we found EHR captured only 21.0-28.1% of all the non-fatal outcomes in the low EHR-continuity cohort but 55.4-66.1% of that in the high EHR-continuity cohort. In the validation set, the best-performing model developed among high EHR-continuity patients had consistently higher AUROC than that based on low-continuity patients: AUROC was 0.849 vs. 0.743 when predicting mortality; AUROC was 0.802 vs. 0.659 predicting the CV events; AUROC was 0.635 vs. 0.567 predicting major bleeding. We observed a similar pattern when using AUPRC as the outcome metric. CONCLUSIONS: Among patients with CV comorbidities, when predicting mortality, major CV events, and bleeding outcomes, the prediction models developed in datasets with low EHR-continuity consistently had worse performance compared to models developed with high EHR-continuity.