Abstract
BACKGROUND: Drug combination therapy often outperforms monotherapy in cancer treatment, but the vast number of available drugs makes manual screening for synergistic combinations costly. Computational methods, especially deep learning, can reduce the search space by predicting likely synergistic drug combinations. Recent studies have improved drug synergy prediction by modeling associations among different biological entities, but drug-drug interactions have not been fully leveraged in this scenario, which motivated the work presented in this paper. METHODS: This paper proposes a deep learning method named HGTSynergy to predict synergistic drug combinations, which employs a heterogeneous graph attention network and a tailored task to capture complex latent patterns in the drug network as prior knowledge. The learned knowledge is then transferred through a transfer learning framework to the downstream task of predicting drug synergy scores, effectively enhancing predictive performance. RESULTS: A five-fold nested cross-validation is employed to train HGTSynergy. In the synergy regression task, HGTSynergy outperforms seven deep learning methods, achieving a mean squared error of 222.83, root mean squared error of 14.91, and Pearson correlation coefficient of 0.75. For the synergy classification task, it also surpasses other methods with an area under the receiver operating characteristic curve of 0.90, area under the precision-recall curve of 0.63, accuracy of 0.94, precision of 0.72, and Cohen's Kappa of 0.52. The ablation study verifies that the heterogeneous graph attention network and the transfer learning framework both have a positive effect on prediction performance. Moreover, a series of analyses demonstrates that the proposed method exhibits strong generalization performance and interpretability. The case study further validates its consistency with prior research. CONCLUSIONS: This study suggests that drug synergy prediction can be improved by comprehensively modeling diverse drug-drug interaction types and leveraging transfer learning to extract prior knowledge from them. The ability of HGTSynergy to discover new anticancer synergistic drug combinations outperforms other state-of-the-art methods. HGTSynergy promises to be a powerful tool to pre-screen anticancer synergistic drug combinations.