Pose Estimation and Non-rigid Registration for Augmented Reality during Neurosurgery - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles IEEE Transactions on Biomedical Engineering Year : 2022

Pose Estimation and Non-rigid Registration for Augmented Reality during Neurosurgery


Objective: A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. Methods: Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a nonrigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. Results: We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. Conclusion: The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. Significance: This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.
Fichier principal
Vignette du fichier
main.pdf (16.25 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03675005 , version 1 (22-05-2022)



Nazim Haouchine, Parikshit Juvekar, Michael Nercessian, William Wells, Alexandra Golby, et al.. Pose Estimation and Non-rigid Registration for Augmented Reality during Neurosurgery. IEEE Transactions on Biomedical Engineering, 2022, 69 (4), pp.1310 - 1317. ⟨10.1109/TBME.2021.3113841⟩. ⟨hal-03675005⟩
112 View
174 Download



Gmail Facebook X LinkedIn More