Abstract
IMPORTANCE: Accurate identification of hospital readmission risk during a current hospitalization may enhance decision-making, facilitate targeted systems-level interventions, and avoid preventable readmissions. OBJECTIVE: To temporally and externally validate a suite of readmission risk prediction models across 48 children's hospitals to assess their generalizability and feasibility for future clinical implementation. DESIGN, SETTING, AND PARTICIPANTS: This prognostic study analyzed data from the Pediatric Health Information System (PHIS) database, which contains billing and resource use data from 48 US children's hospitals, including the derivation hospital (DH) and 47 hospitals participating in the PHIS database (hereafter other PHIS hospitals). Children aged 18 years or younger discharged from these hospitals between January 1, 2016, to December 31, 2019, were included. This cohort was divided as specified into the 3 prediction models at the DH: 6 months or older with no recent hospitalizations (new admission model [NAM]), 6 months or older with 1 or more prior hospitalizations within the last 6 months (recent admission model [RAM]), and 6 months or younger (young infant model [YIM]). Data were analyzed from August 9 to December 1, 2023. MAIN OUTCOMES AND MEASURES: The primary validation outcome was hospital-level discrimination measured with area under the receiver operating characteristic curve (AUROC). Predictors included demographic, clinical, and utilization variables. All-cause 30-day readmission was modeled for each hospital using logistic regression and parameter estimates from the DH. Calibration plots examined observed vs predicted outcome frequencies for each hospital. RESULTS: In external validation, a total of 851 499 children were discharged from 48 hospitals (16 330 DH discharges and 835 169 other PHIS hospital discharges). The largest group of children was aged 5 to 14 years (281 193 [33.0%]). In temporal validation, the DH PHIS 2016-2018 cohort included 45 682 discharges. All-cause 30-day readmission rates were 7.2% for NAM, 35.5% for RAM, and 11.7% for YIM. The 2019 DH PHIS cohort included 16 330 discharges. All cause 30-day readmision rates were 7.2% for NAM, 35.1% for RAM, and 11.1% for YIM. Temporal validation demonstrated reduced discrimination across all 3 models (median AUROC, 0.65 [95% CI 0.62-0.67] for the NAM; 0.73 [95% CI 0.72-0.75) for RAM; 0.67 [95% CI 0.63-0.70) for the YIM compared with the original estimates (median AUROC 0.76 [95% CI 0.85-0.78] for the NAM; 0.84 [95% CI 0.83-0.84] for the RAM; 0.79 [95% CI 0.77-0.80] for the YIM). Overall readmission rates were 5.9% for NAM, 30.1% for RAM, and 7.6% for YIM. External validation yielded similiar findings as the temporal validation, although with demonstrable variation in performance across hospitals (median [range] AUROC, 0.64 [0.60-0.68] for the NAM; 0.73 [0.64-0.80] for the RAM; 0.65 [0.53-0.74] for the YIM). Most hospitals were poorly calibrated, with both significant overestimation and underestimation of observed risk. Of 47 other PHIS hospitals, only 3 for the RAM (6.4%) and 9 for both the NAM and YIM (19.1%) were adequately calibrated. CONCLUSIONS AND RELEVANCE: This prognostic study found that the readmission risk prediction models had reduced predictive accuracy across time and variability in hospital-level performance. These findings stress the importance of local validation prior to clinical implementation and suggest opportunities to improve generalizability, including multicenter derivation and expansion of candidate predictors.