Abstract
Advancements in deep learning technologies and an increase in medical data have enhanced the accuracy of disease diagnosis and treatment strategies. Notably, significant progress has been made in the use of deep learning-based time-series prediction models for short-term disease onset prediction and analysis of important features. However, research on explainable deep learning for long-term disease progression, such as cancer and chronic diseases, still faces challenges. The difficulty in estimating explainable gradual disease progression from observable patient test data is a key factor. To address this issue, we propose a new approach called the "deep state-space analysis framework." This framework utilizes sequentially obtained electronic health records (EHRs) to estimate and visualize temporal changes in the latent states of patients related to disease progression. It enables the clustering of latent patient states according to the severity of disease progression and identifies key factors leading to a poor prognosis with medication. To validate our framework, a detailed analysis of data from 12,695 patients with cancer was conducted. The estimated transitions of the latent states capture the clinical status of the patients and their continuous temporal changes. Furthermore, anemia was identified as a poor prognostic factor during state transitions in patients with cancer. Significant features were also confirmed, such as immune cell abnormalities, which are poor prognostic factors in patients treated with Nivolumab, Osimertinib, and Afatinib. This technological innovation deepens our understanding of disease progression and supports early treatment adjustments, prognostic evaluations, and the formulation of optimal long-term strategies. With the advancements in deep learning, its application in healthcare has even greater potential.