Abstract
BACKGROUND: The fusion of intraoperative 2D laparoscopic images with preoperative 3D scans offers significant advantages in minimally invasive surgery, such as improved spatial understanding and enhanced navigation. This study aims to enable augmented reality for deformable organs through accurate 2D-3D registration. However, achieving real-time and precise alignment remains a major challenge due to organ deformation, occlusion, and the difficulty of estimating camera parameters from monocular images. METHODS: We introduce a non-rigid image-volume registration (NRIVR) framework designed specifically for deformable human organs. Our approach employs a long short-term memory-based camera estimation neural network (LCENN) to predict camera poses directly from 2D anatomical contours extracted from laparoscopic images. By leveraging a differentiable mapping from 2D boundaries to camera parameters, the system enables real-time inference. Non-rigid registration is then performed in 2D space by integrating both the projected mesh and estimated deformation fields, ensuring consistent alignment across views. RESULTS: Our experiments, evaluating the contour mapping neural network on laparoscopic images from cholecystectomy, showed that using an LCENN can efficiently predict the camera pose from 2D boundaries, achieving a minimal rotational error of 0.35±0.44° and a translational error of 0.51±0.31 mm. Consequently, our proposed framework effectively achieved 2D-3D registration on a clinical dataset, with an average target registration error of 2.74±1.51 mm. CONCLUSIONS: These results validate the feasibility and effectiveness of the proposed method for real-time 2D-3D registration in laparoscopic surgery, paving the way for enhanced image guidance in clinical workflows.