Abstract
In medical imaging, deep learning (DL) models often struggle to delineate ambiguous structures such as tumors or organ boundaries, leading to uncertainty in defining precise contours. This challenge is amplified by inter-rater variability, where experts may disagree on boundary delineations, resulting in inconsistent segmentation outcomes. Addressing these issues requires robust algorithms capable of quantifying uncertainty, standardizing annotation practices, and improving calibration to ensure reliable predictions, particularly in multi-class and multi-rater scenarios. When models are miscalibrated and overconfident, their outputs can mislead clinical decision-making, potentially influencing radiologists to over- or under-estimate malignancy risks. The CURVAS challenge (Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation) was established to address these challenges by jointly assessing uncertainty, calibration, and segmentation quality, as well as promoting clinical relevance by evaluating organ volumes while accounting for annotation variability. To support this, a dataset of 90 contrast-enhanced CT scans from University Hospital Erlangen was curated, containing pancreas, liver, and kidney segmentations annotated by three experts. This resource provides a foundation for developing and benchmarking algorithms that balance segmentation accuracy, calibration, and reliability. A quantitative analysis of the annotations shows that kidney and liver segmentations exhibit strong consistency, whereas the pancreas remains challenging, emphasizing the need for refined labeling protocols and improved training strategies.