Abstract
The development of artificial intelligence-based imaging techniques for prostate cancer (PCa) detection and diagnosis requires a reliable ground truth, which is generally based on histopathology from radical prostatectomy specimens. This study proposes a comprehensive protocol for the annotation of prostatectomy pathology slides. To evaluate the reliability of the protocol, interobserver variability was assessed between five pathologists, who annotated ten radical prostatectomy specimens consisting of 74 whole mount pathology slides. Interobserver variability was assessed for both the localization and grading of PCa. The results indicate excellent overall agreement on the localization of PCa (Gleason pattern ≥ 3) and clinically significant PCa (Gleason pattern ≥ 4), with Dice similarity coefficients (DSC) of 0.91 and 0.88, respectively. On a per-slide level, agreement for primary and secondary Gleason pattern was almost perfect and substantial, with Fleiss Kappa of .819 (95% CI .659-.980) and .726 (95% CI .573-.878), respectively. Agreement on International Society of Urological Pathology Grade Group was evaluated for the index lesions and showed agreement in 70% of cases, with a mean DSC of 0.92 for all index lesions. These findings show that a standardized protocol for prostatectomy pathology annotation provides reliable data on PCa localization and grading, with relatively high levels of interobserver agreement. More complicated tissue characterization, such as the presence of cribriform growth and intraductal carcinoma, remains a source of interobserver variability and should be treated with care when used in ground truth datasets.