Anterior vertebral tethering (AVT) is a non-invasive spine surgery technique, treating severe spine deformations and preserving lower back mobility. However, patient positioning and surgical strategies greatly influences postoperative results. Predicting the upright geometry from pediatric spines is needed to optimize patient positioning in the operating room (OR) and improve surgical outcomes, but remains a complex task due to immature bone properties. We propose a framework used in the OR predicting the upright spine geometry at the first visit following surgery in idiopathic scoliosis patients. The approach first creates a 3D model of the spine while the patient is on the operating table. For this, multiview Transformers that combine images from different viewpoints are used to generate the intraoperative pose. The postoperative upright shape is then predicted on-the-fly using implicit neural fields, which are trained from geometries at different time points and conditioned with surgical parameters. A Signed Distance Function for shape constellations is used to handle the variability in spine appearance, capturing a disentangled latent domain of the articulation vectors, with separate encoding vectors representing both articulation and shape parameters. A regularization criterion based on a pre-trained group-wise trajectory of spine transformations generates complete spine models. A training set of 652 patients with 3D models was used to train the model, tested on a distinct cohort of 83 surgical patients. The framework based on neural kernels predicted upright 3D geometries with a mean 3D error of 1 . 3 ± 0 . 5 mm in landmarks points, and IoU of 95.9% in vertebral shapes when compared to actual postop models, falling within the acceptable margins of error below 2 mm. • We introduce a forecasting method of the standing spine shape during surgery. • An articulated neural kernel field disentangles spine’s latent representations. • An articulation/shape network is used to generate the standing shape. • Multi-view model based on view-divergent Transformers infers intra-op 3D model. • Clinical efficacy is demonstrated in spine surgery plans with several deformations.

In minimally invasive procedures such as biopsies and prostate cancer brachytherapy, accurate needle placement remains challenging due to limitations in current tracking methods related to interference, reliability, resolution or image contrast. This often leads to frequent needle adjustments and reinsertions. To address these shortcomings, we introduce an optimized needle shape-sensing method using a fully distributed grating-based sensor. The proposed method uses simple trigonometric and geometric modeling of the fiber using optical frequency domain reflectometry (OFDR), without requiring prior knowledge of tissue properties or needle deflection shape and amplitude. Our optimization process includes a reproducible calibration process and a novel tip curvature compensation method. We validate our approach through experiments in artificial isotropic and inhomogeneous animal tissues, establishing ground truth using 3D stereo vision and cone beam computed tomography (CBCT) acquisitions, respectively. Our results yield an average RMSE ranging from 0.58 ± 0.21 mm to 0.66 ± 0.20 mm depending on the chosen spatial resolution, achieving the submillimeter accuracy required for interventional procedures.

Deep neural networks are commonly used for automated medical image segmentation, but models will frequently struggle to generalize well across different imaging modalities. This issue is particularly problematic due to the limited availability of annotated data, both in the target as well as the source modality, making it difficult to deploy these models on a larger scale. To overcome these challenges, we propose a new semi-supervised training strategy called MoDATTS. Our approach is designed for accurate cross-modality 3D tumor segmentation on unpaired bi-modal datasets. An image-to-image translation strategy between modalities is used to produce synthetic but annotated images and labels in the desired modality and improve generalization to the unannotated target modality. We also use powerful vision transformer architectures for both image translation (TransUNet) and segmentation (Medformer) tasks and introduce an iterative self-training procedure in the later task to further close the domain gap between modalities, thus also training on unlabeled images in the target modality. MoDATTS additionally allows the possibility to exploit image-level labels with a semi-supervised objective that encourages the model to disentangle tumors from the background. This semi-supervised methodology helps in particular to maintain downstream segmentation performance when pixel-level label scarcity is also present in the source modality dataset, or when the source dataset contains healthy controls. The proposed model achieves superior performance compared to other methods from participating teams in the CrossMoDA 2022 vestibular schwannoma (VS) segmentation challenge, as evidenced by its reported top Dice score of 0.87±0.04 for the VS segmentation. MoDATTS also yields consistent improvements in Dice scores over baselines on a cross-modality adult brain gliomas segmentation task composed of four different contrasts from the BraTS 2020 challenge dataset, where 95% of a target supervised model performance is reached when no target modality annotations are available. We report that 99% and 100% of this maximum performance can be attained if 20% and 50% of the target data is additionally annotated, which further demonstrates that MoDATTS can be leveraged to reduce the annotation burden.

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

Real-time motion management for image-guided radiation therapy interventions plays an important role for accurate dose delivery. Forecasting future 4D deformations from in-plane image acquisitions is fundamental for accurate dose delivery and tumor targeting. However, anticipating visual representations is challenging and is not exempt from hurdles such as the prediction from limited dynamics, and the high-dimensionality inherent to complex deformations. Also, existing 3D tracking approaches typically need both template and search volumes as inputs, which are not available during real-time treatments. In this work, we propose an attention-based temporal prediction network where features extracted from input images are treated as tokens for the predictive task. Moreover, we employ a set of learnable queries, conditioned on prior knowledge, to predict future latent representation of deformations. Specifically, the conditioning scheme is based on estimated time-wise prior distributions computed from future images available during the training stage. Finally, we propose a new framework to address the problem of temporal 3D local tracking using cine 2D images as inputs, by employing latent vectors as gating variables to refine the motion fields over the tracked region. The tracker module is anchored on a 4D motion model, which provides both the latent vectors and the volumetric motion estimates to be refined. Our approach avoids auto-regression and leverages spatial transformations to generate the forecasted images. The tracking module reduces the error by 63% compared to a conditional-based transformer 4D motion model, yielding a mean error of 1.5± 1.1 mm. Furthermore, for the studied cohort of abdominal 4D MRI images, the proposed method is able to predict future deformations with a mean geometrical error of 1.2± 0.7 mm.

Recent fusionless surgical techniques for corrective spine surgery such as Anterior Vertebral Body Growth Modulation (AVBGM) allow to treat mild to severe spinal deformations by tethering vertebral bodies together, helping to preserve lower back flexibility. Forecasting the outcome of AVBGM from skeletally immature patients remains elusive with several factors involved in corrective vertebral tethering, but could help orthopaedic surgeons plan and tailor AVBGM procedures prior to surgery. We introduce an intra-operative framework forecasting the outcomes during AVBGM surgery in scoliosis patients. The method is based on spatial-temporal corrective networks, which learns the similarity in segmental corrections between patients and integrates a long-term shifting mechanism designed to cope with timing differences in onset to surgery dates, between patients in the training set. The model captures dynamic geometric dependencies in scoliosis patients, ensuring long-term dependency with temporal dynamics in curve evolution and integrated features from inter-vertebral disks extracted from T2-w MRI. The loss function of the network introduces a regularization term based on learned group-average piecewise-geodesic path to ensure the generated corrective transformations are coherent with regards to the observed evolution of spine corrections at follow-up exams. The network was trained on 695 3D spine models and tested on 72 operative patients using a set of 3D spine reconstructions as inputs. The spatio-temporal network predicted outputs with errors of 1.8 ± 0.8mm in 3D anatomical landmarks, yielding geometries similar to ground-truth spine reconstructions obtained at one and two year follow-ups and with significant improvements to comparative deep learning and biomechanical models.

Flexible medical instruments, such as Continuum Dexterous Manipulators (CDM), constitute an important class of tools for minimally invasive surgery. Accurate CDM shape reconstruction during surgery is of great importance, yet a challenging task. Fiber Bragg grating (FBG) sensors have demonstrated great potential in shape sensing and consequently tip position estimation of CDMs. However, due to the limited number of sensing locations, these sensors can only accurately recover basic shapes, and become unreliable in the presence of obstacles or many inflection points such as s-bends. Optical Frequency Domain Reflectometry (OFDR), on the other hand, can achieve much higher spatial resolution, and can therefore accurately reconstruct more complex shapes. Additionally, Random Optical Gratings by Ultraviolet laser Exposure (ROGUEs) can be written in the fibers to increase signal to noise ratio of the sensors. In this comparison study, the tip position error is used as a metric to compare both FBG and OFDR shape reconstructions for a 35 mm long CDM developed for orthopedic surgeries, using a pair of stereo cameras as ground truth. Three sets of experiments were conducted to measure the accuracy of each technique in various surgical scenarios. The tip position error for the OFDR (and FBG) technique was found to be 0.32 (0.83) mm in free-bending environment, 0.41 (0.80) mm when interacting with obstacles, and 0.45 (2.27) mm in s-bending. Moreover, the maximum tip position error remains sub-millimeter for the OFDR reconstruction, while it reaches 3.40 mm for FBG reconstruction. These results propose a cost-effective, robust and more accurate alternative to FBG sensors for reconstructing complex CDM shapes.

In brain tumor surgery, the quality and safety of the procedure can be impacted by intra-operative tissue deformation, called brain shift. Brain shift can move the surgical targets and other vital structures such as blood vessels, thus invalidating the pre-surgical plan. Intra-operative ultrasound (iUS) is a convenient and cost-effective imaging tool to track brain shift and tumor resection. Accurate image registration techniques that update pre-surgical MRI based on iUS are crucial but challenging. The MICCAI Challenge 2018 for Correction of Brain shift with Intra-Operative UltraSound (CuRIOUS2018) provided a public platform to benchmark MRI-iUS registration algorithms on newly released clinical datasets. In this work, we present the data, setup, evaluation, and results of CuRIOUS 2018, which received 6 fully automated algorithms from leading academic and industrial research groups. All algorithms were first trained with the public RESECT database, and then ranked based on a test dataset of 10 additional cases with identical data curation and annotation protocols as the RESECT database. The article compares the results of all participating teams and discusses the insights gained from the challenge, as well as future work.

Intra-arterial liver cancer therapies, such as trans-arterial chemoembolization, are the preferred therapeutic approaches for advanced hepatocellular carcinoma. However, these palliative techniques are challenging for delivering therapeutic agents selectively in the tumor without real-time 3-D visualization of the catheter within the hepatic arteries. The objective of this paper is to develop and evaluate in pre-clinical tests an advanced interventional guidance platform using a distributed strain sensing device based on optical frequency-domain reflectometry (OFDR) to track the tip and shape of a catheter. The scattering properties of a fiber triplet are enhanced by focusing an ultraviolet beam on these fibers, producing a fully distributed strain sensor, which avoids interpolation errors observed with traditional shape tracking systems. A 3-D roadmap of the hepatic arteries is obtained from a combined fully convolutional and residual networks trained on MR angiography and combined with a 4-D flow dynamic sequence enabling to map blood flow velocities. An anisotropic curvature matching method is proposed to map the sensed data onto pre-operative MR and using 3-D ultrasound to correct for non-rigid deformations. Experiments were conducted in a controlled environment setting as well as in both synthetic phantoms and in five porcine models to assess the performance for device navigation, yielding satisfactory tracking accuracy with 3-D mean errors of 2.8 ± 0.9 mm. We present the first pilot study of MR-compatible UV-exposed OFDR optical fibers for non-ionizing device guidance in intra-arterial procedures, with the potential of avoiding multiple hospitalizations required to perform invasive selective chemoembolizations.

It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property. This paper explores issues with optimization convergence, speed and gradient stability when encouraging or enforcing orthogonality. To perform this analysis, we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation. We find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance.

We introduce a novel approach for predicting the progression of adolescent idiopathic scoliosis from 3-D spine models reconstructed from biplanar X-ray images. Recent progress in machine learning has allowed to improve classification and prognosis rates, but lack a probabilistic framework to measure uncertainty in the data. We propose a discriminative probabilistic manifold embedding where locally linear mappings transform data points from high-dimensional space to corresponding low-dimensional coordinates. A discriminant adjacency matrix is constructed to maximize the separation between progressive (P) and nonprogressive (NP) groups of patients diagnosed with scoliosis, while minimizing the distance in latent variables belonging to the same class. To predict the evolution of deformation, a baseline reconstruction is projected onto the manifold, from which a spatiotemporal regression model is built from parallel transport curves inferred from neighboring exemplars. Rate of progression is modulated from the spine flexibility and curve magnitude of the 3-D spine deformation. The method was tested on 745 reconstructions from 133 subjects using longitudinal 3-D reconstructions of the spine, with results demonstrating the discriminatory framework can identify between P and NP of scoliotic patients with a classification rate of 81% and the prediction differences of 2.1° in main curve angulation, outperforming other manifold learning methods. Our method achieved a higher prediction accuracy and improved the modeling of spatiotemporal morphological changes in highly deformed spines compared with other learning methods.

Quantifying spinal cord (SC) atrophy in neurodegenerative and traumatic diseases brings important diagnosis and prognosis information for the clinician. We recently developed the PropSeg method, which allows for fast, accurate and automatic segmentation of the SC on different types of MRI contrast (e.g., T1-, T2- and T2(∗) -weighted sequences) and any field of view. However, comparing measurements from the SC between subjects is hindered by the lack of a generic coordinate system for the SC. In this paper, we present a new framework combining PropSeg and a vertebral level identification method, thereby enabling direct inter- and intra-subject comparison of SC measurements for large cohort studies as well as for longitudinal studies. Our segmentation method is based on the multi-resolution propagation of tubular deformable models. Coupled with an automatic intervertebral disk identification method, our segmentation pipeline provides quantitative metrics of the SC and spinal canal such as cross-sectional areas and volumes in a generic coordinate system based on vertebral levels. This framework was validated on 17 healthy subjects and on one patient with SC injury against manual segmentation. Results have been compared with an existing active surface method and show high local and global accuracy for both SC and spinal canal (Dice coefficients =0.91 ± 0.02) segmentation. Having a robust and automatic framework for SC segmentation and vertebral-based normalization opens the door to bias-free measurement of SC atrophy in large cohorts.

International audience

This paper presents a novel 3-D reconstruction method of the scoliotic spine using prior vertebra models with image-based information taken from biplanar X-ray images. We first propose a global modeling approach by exploiting the 3-D scoliotic curve reconstructed from a coronal and sagittal X-ray image in order to generate an approximate statistical model from a 3-D database of scoliotic patients based on a transformation algorithm which incorporates intuitive geometrical properties. The personalized 3-D reconstruction of the spine is then achieved with a novel segmentation method which takes into account the variable appearance of scoliotic vertebrae (rotation, wedging) from standard quality images in order to segment and isolate individual vertebrae on the radiographic planes. More specifically, it uses prior 3-D models regulated from 2-D image level set functionals to identify and match corresponding bone structures on the biplanar X-rays. An iterative optimization procedure integrating similarity measures such as deformable vertebral contours regulated from high-level anatomical primitives, morphological knowledge and epipolar constraints is then applied to globally refine the 3-D anatomical landmarks on each vertebra level of the spine. This method was validated on twenty scoliotic patients by comparing results to a standard manual approach. The qualitative evaluation of the retro-projection of the vertebral contours confirms that the proposed method can achieve better consistency to the X-ray image's natural content. A comparison to synthetic models and real patient data also yields good accuracy on the localization of low-level primitives such as anatomical landmarks identified by an expert on each vertebra. The experiments reported in this paper demonstrate that the proposed method offers a better matching accuracy on a set of landmarks from biplanar views when compared to a manual technique for each evaluated cases, and its precision is comparable to 3-D models generated from magnetic resonance images, thus suitable for routine 3-D clinical assessment of spinal deformities.