Automated 3D liver segmentation and mixed reality integration for preoperative surgical planning

Abstract

Traditional liver surgical planning, relying on manual interpretation of 2D images, is often limited in precision and efficiency. This report introduces an integrated system designed to revolutionize this practice by combining automated deep learning-based liver segmentation with a collaborative Mixed Reality (MR) environment. The developed approach leverages advanced neural network architectures for accurate liver and tumor segmentation, followed by rapid 3D reconstruction. The segmentation model achieved a median Dice score of 0.92 on the IRCAD dataset, with comparable performance on LiTS and MDHV. The generated 3D models are then imported into an interactive MR application, enabling immersive visualization and intuitive manipulation. Furthermore, the system supports multiple simultaneous users in local collaborative mode, facilitating joint discussion and planning. This unique contribution, merging automated segmentation with immersive MR collaboration, significantly enhances the precision and efficiency of surgical planning, offering substantial potential for improving clinical outcomes. The emphasis on these key figures and the system’s unique contribution highlights that the project’s value lies not only in the performance of its individual components but also in the synergy created by integrating AI and MR to optimize a complex clinical workflow.