Abstract
Thyroid cancer recurrence poses a significant challenge in clinical practice, often complicating treatment outcomes and long-term patient management. This study investigates the application of multiple machine learning models-Logistic Regression, Decision Trees, Random Forest, Gradient Boosting, Support Vector Machines (SVM), and K-Nearest Neighbors (KNN)-to predict recurrence using a real-world dataset of 383 patients. Among the evaluated models, the Random Forest classifier achieved the highest accuracy of 98.26%, demonstrating a strong balance between sensitivity and specificity. To mitigate class imbalance, class weights were incorporated into the model training. Model performance was assessed through Stratified 5-Fold Cross-Validation. For robust hyperparameter tuning and a reliable performance estimate, we implemented nested cross-validation with GridSearchCV. Analysis of learning curves provided insights into the model's behavior with varying training data sizes. Furthermore, we assessed and improved the reliability of the model's predicted probabilities through model calibration using reliability curves and Isotonic Regression, which is crucial for clinical decision-making. The model demonstrated promising performance (mean nested CV accuracy: ~0.964), and the calibration significantly improved the trustworthiness of probability estimates. Comprehensive preprocessing and feature selection methods, including Chi-square and Random Forest Gini Importance, were applied to enhance model performance. Further explainability was added using SHAP analysis to understand key feature contributions. The findings underscore the potential of machine learning-based frameworks to support early identification of high-risk recurrence cases, thereby assisting clinicians in tailoring follow-up and treatment strategies more effectively.