Abstract
Background/Objectives: To simplify the decision-making process in radiomics by employing RadiomiX, an algorithm designed to automatically identify the best model combination and validate them across multiple environments was developed, thus enhancing the reliability of results. Methods: RadiomiX systematically tests classifier and feature selection method combinations known to be suitable for radiomic datasets to determine the best-performing configuration across multiple train-test splits and K-fold cross-validation. The framework was validated on four public retrospective radiomics datasets including lung nodules, metastatic breast cancer, and hepatic encephalopathy using CT, PET/CT, and MRI modalities. Model performance was assessed using the area under the receiver-operating-characteristic curve (AUC) and accuracy metrics. Results: RadiomiX achieved superior performance across four datasets: LLN (AUC = 0.850 and accuracy = 0.785), SLN (AUC = 0.845 and accuracy = 0.754), MBC (AUC = 0.889 and accuracy = 0.833), and CHE (AUC = 0.837 and accuracy = 0.730), significantly outperforming original published models (p < 0.001 for LLN/SLN and p = 0.023 for MBC accuracy). When original published models were re-evaluated using ten-fold cross-validation, their performance decreased substantially: LLN (AUC = 0.783 and accuracy = 0.731), SLN (AUC = 0.748 and accuracy = 0.714), MBC (AUC = 0.764 and accuracy = 0.711), and CHE (AUC = 0.755 and accuracy = 0.677), further highlighting RadiomiX's methodological advantages. Conclusions: Systematically testing model combinations using RadiomiX has led to significant improvements in performance. This emphasizes the potential of automated ML as a step towards better-performing and more reliable radiomic models.