Abstract
BACKGROUND: The extraction of overlapping cell nuclei is a critical issue in automated diagnosis systems. Due to the similarities between overlapping and malignant nuclei, misclassification of the overlapped regions can affect the automated systems' final decision. In this paper, we present a method for detecting overlapping cell nuclei in Pap smear samples. METHOD: Judgement about the presence of overlapping nuclei is performed in three steps using an unsupervised clustering approach: candidate nuclei regions are located and refined with morphological operations; key features are extracted; and candidate nuclei regions are clustered into two groups, overlapping or non-overlapping, A new combination of features containing two local minima-based and three shape-dependent features are extracted for determination of the presence or absence of overlapping. F1 score, precision, and recall values are used to evaluate the method's classification performance. RESULTS: In order to make evaluation, we compared the segmentation results of the proposed system with empirical contours. Experimental results indicate that applied morphological operations can locate most of the nuclei and produces accurate boundaries. Independent features significance test indicates that our feature combination is significant for overlapping nuclei. Comparisons of the classification results of a fuzzy clustering algorithm and a non-fuzzy clustering algorithm show that the fuzzy approach would be a more convenient mechanism for classification of overlapping. CONCLUSION: The main contribution of this study is the development of a decision mechanism for identifying overlapping nuclei to further improve the extraction process with respect to the segmentation of interregional borders, nuclei area, and radius. Experimental results showed that our unsupervised approach with proposed feature combination yields acceptable performance for detection of overlapping nuclei.