Scalable VR & AR Technologies for Surgical Planning in Clinical Medicine

Abstract

The surgical planning landscape is undergoing a significant transformation due to technological advancements and the increasing complexity of surgical interventions. The integration of scalable Virtual Reality (VR) systems into surgical planning has emerged as a key innovation. Traditionally, surgical planning depends on imaging techniques like X-rays, CT scans, and MRI, offering a limited perspective due to their two-dimensional nature. Surgeons often have to mentally reconstruct a threedimensional understanding from these two-dimensional images, a process that can lead to errors. Further, these methods struggle to accurately represent the spatial relationship between critical structures and variations in patient anatomy, and do not allow for pre-surgical simulations. VR technology offers an immersive, interactive 3D environment that closely mimics the surgical field, enabling surgeons to visualise and interact with patient-specific anatomical models derived from actual patient imaging data. This enhances understanding of complex anatomical relationships and facilitates precise surgical planning. VR’s potential to revolutionise surgical planning and execution is further supported by the increase in data collected before a surgical procedure.

Previous research demonstrated the positive contribution VR can offer in surgical planning in various medical fields, such as chest wall resections, oral and maxillofacial surgery, robotic-assisted partial nephrectomy, tracheoplasty, and cranial aneurysm clipping. Although these implementations are functional and contributed great knowledge to the field, adoption of these VR surgical planning tools has struggled with scalability and validation in real clinical environments. Often- times these systems are built upon ad hoc platforms and are not a part of a medical data acquisition and processing pipeline and as such cannot be adapted to different types of operations and, therefore, cannot be scaled and reused into other disciplines. Most of previous research also lacks clinical deployment on a day-to-day basis and validation with real-patient cases, mostly due to high friction to change clinical data acquisition protocols and regulatory affairs. Making matters worse, current VR solutions, both research and commercial, have non-standardised user interaction interfaces and paradigms between different operation types to navigate medical visualisations, preventing wider adoption by heavily time-constrained clinical experts. Finally, there is a need to establish a -secure- system for the distribution of data required to generate 3D-VR-ready reconstruction of patient models, akin to the functionality offered by Picture Archiving and Communication Systems (PACS) for the distribution of standard medical imaging.

In this thesis, in an attempt to improve upon the aforementioned issues, we first introduce IKEM VRLab. The system represents a modular and extensible VR platform designed from the bottom-up to integrate in a medical data pipeline for clinical deployment. Developed with a focus on scalability and adaptability, IKEM VRLab facilitates immersive, interactive examinations of patient-specific anatomical data in 3D. IKEM VRLab supports the implementation of specialised toolkits for a variety of surgical procedures, enabling VR technologies to pervade in various clinical settings. We extensively validated our system with a novel method to examine the volumetric accuracy of CT-derived VR-optimised 3D models. The results indicated that there was no statistically significant difference between the volume estimated using IKEM VRLab and the actual organ volume, which is crucial for pre-operative planning.

Building upon the capabilities of IKEM VRLab, we then introduce OrthopedVR as the first specialised application toolkit for the planning and visualisation of corrective surgeries for lower limb abnormalities, such as rotational deformities causing patellar mal-tracking. This application allows orthopaedic surgeons to simulate derotational osteotomies in a 3D environment, offering a more intuitive understanding of patient anatomy and enhancing surgical precision. The evaluation of OrthopedVR by experienced surgeons highlights its superiority over traditional 2D imaging techniques, indicating its potential to significantly improve pre-operative planning, surgical outcomes, and clinical training.

We then further expand the scope of IKEM VRLab, exploring its application in liver resection planning, a critical aspect of treating primary and secondary liver cancers as well as echinococcal cysts. The VRLab toolkit for liver resections enables detailed visualisation and interaction with 3D reconstructions of the liver, facilitating accurate and personalised surgical planning. Through immersive VR, surgeons demonstrably achieved a deeper comprehension of complex anatomical structures, improving decision-making and potentially reducing post-operative complications. Evaluation of the toolkit involving experienced liver surgeons confirmed the system’s effectiveness in enhancing the accuracy and efficacy of liver surgeries compared to conventional medical visualisation standards.

We finally shift our attention to lung surgeries. We first extend IKEM VRLab to support Augmented Reality (AR) to assist with planning port placements for Robotic-Assisted Thoracic Surgery (RATS) during lung segmentectomies. This AR Lung toolkit aims to minimise the invasive impact of RATS by enhancing the visualisation and planning of trocars and instrument placements. We then focus on lung transplant, attempting organ transplant size matching. IKEM VRLab’s Fitting Room offers an innovative approach to donor-recipient child-adult size-matching, providing surgeons with an immersive platform to assess and optimise lung transplant compatibility. The lung toolkit demonstrates VR’s potential to improve the precision of organ matching and lung surgeries, contributing to better post-operative outcomes and patient survival rates.

The integration of scalable VR and AR systems in surgical planning, as achieved by IKEM VRLab, marks a significant advancement in the field. This thesis demonstrates that VR technologies, by providing an immersive and interactive 3D environment, enhance the understanding of complex anatomical structures and improve the precision of surgical interventions across various medical fields. The modular and adaptable nature of IKEM VRLab allows it to integrate seamlessly into medical data pipelines, supporting a range of specialized surgical procedures. Evaluations of specific toolkits for orthopaedic, liver, and lung surgeries validate the system’s effectiveness in clinical settings, indicating its potential to improve pre-operative planning, surgical outcomes, and clinical training. This work also underscores the importance of standardizing VR interfaces and ensuring secure data distribution to foster wider adoption and scalability in clinical environments.