Abstract
If we do not urgently educate current and future medical professionals to critically evaluate and distinguish credible AI-assisted diagnostic tools from those whose performance is artificially inflated by data leakage or improper validation, we risk undermining clinician trust in all AI diagnostics and jeopardizing future advances in patient care. For instance, machine learning models have shown high accuracy in diagnosing Parkinson's Disease when trained on clinical features that are themselves diagnostic, such as tremor and rigidity. This study systematically investigates the impact of data leakage and feature selection on the true clinical utility of machine learning models for early Parkinson's Disease detection. We constructed two experimental pipelines: one excluding all overt motor symptoms to simulate a subclinical scenario and a control including these features. Nine machine learning algorithms were evaluated using a robust three-way data split and comprehensive metric analysis. Results reveal that, without overt features, all models exhibited superficially acceptable F1 scores but failed catastrophically in specificity, misclassifying most healthy controls as Parkinson's Disease. The inclusion of overt features dramatically improved performance, confirming that high accuracy was due to data leakage rather than genuine predictive power. These findings underscore the necessity of rigorous experimental design, transparent reporting, and critical evaluation of machine learning models in clinically realistic settings. Our work highlights the risks of overestimating model utility due to data leakage and provides guidance for developing robust, clinically meaningful machine learning tools for early disease detection.