• Demonstration of effectiveness of SSL for surgical computer vision using the largest dataset reported to date. • Strong generalization and robust evaluation are shown across six surgical datasets, four procedures, and three tasks, outperforming current SOTA foundation models. • Providing insights into large-scale SSL for surgical computer vision in terms of scaling, pretraining time, dataset composition, and model architecture. • Release of the models and a curated dataset of 2.1 million surgical video frames, establishing a critical resource for advancing surgical foundation model training Foundation models have revolutionized computer vision by achieving vastly superior performance across diverse tasks through large-scale pretraining on extensive datasets. However, their application in surgical computer vision has been limited. This study addresses this gap by introducing SurgeNetXL, a novel surgical foundation model that sets a new benchmark in surgical computer vision. Trained on the largest reported surgical dataset to date, comprising over 4.7 million video frames, SurgeNetXL achieves consistent top-tier performance across six datasets spanning four surgical procedures and three tasks, including semantic segmentation, surgical phase recognition, and critical view of safety (CVS) classification. Compared with the best-performing surgical foundation model, SurgeNetXL shows mean improvements of 4.0%, 8.9%, and 11.4% for semantic segmentation, phase recognition, and CVS classification, respectively. Additionally, SurgeNetXL outperforms ImageNet1k by 16.1%, 8.0%, and 4.3% for the respective tasks. In addition to advancing model performance, this study provides key insights into scaling pretraining datasets, extending training durations, and optimizing model architectures specifically for surgical computer vision. These findings pave the way for improved generalization and robustness in data-scarce scenarios, offering a comprehensive framework for future research in this domain. All models and a subset of the SurgeNetXL dataset, including over 2 million video frames, are publicly available at: https://github.com/TimJaspers0801/SurgeNet .

Deep learning-based segmentation methods provide an effective and automated way for assessing the structure and function of the heart in cardiac magnetic resonance (CMR) images. However, despite their state-of-the-art performance on images acquired from the same source (same scanner or scanner vendor) as images used during training, their performance degrades significantly on images coming from different domains. A straightforward approach to tackle this issue consists of acquiring large quantities of multi-site and multi-vendor data, which is practically infeasible. Generative adversarial networks (GANs) for image synthesis present a promising solution for tackling data limitations in medical imaging and addressing the generalization capability of segmentation models. In this work, we explore the usability of synthesized short-axis CMR images generated using a segmentation-informed conditional GAN, to improve the robustness of heart cavity segmentation models in a variety of different settings. The GAN is trained on paired real images and corresponding segmentation maps belonging to both the heart and the surrounding tissue, reinforcing the synthesis of semantically-consistent and realistic images. First, we evaluate the segmentation performance of a model trained solely with synthetic data and show that it only slightly underperforms compared to the baseline trained with real data. By further combining real with synthetic data during training, we observe a substantial improvement in segmentation performance (up to 4% and 40% in terms of Dice score and Hausdorff distance) across multiple data-sets collected from various sites and scanner. This is additionally demonstrated across state-of-the-art 2D and 3D segmentation networks, whereby the obtained results demonstrate the potential of the proposed method in tackling the presence of the domain shift in medical data. Finally, we thoroughly analyze the quality of synthetic data and its ability to replace real MR images during training, as well as provide an insight into important aspects of utilizing synthetic images for segmentation.

In recent years, several deep learning models have been proposed to accurately quantify and diagnose cardiac pathologies. These automated tools heavily rely on the accurate segmentation of cardiac structures in MRI images. However, segmentation of the right ventricle is challenging due to its highly complex shape and ill-defined borders. Hence, there is a need for new methods to handle such structure's geometrical and textural complexities, notably in the presence of pathologies such as Dilated Right Ventricle, Tricuspid Regurgitation, Arrhythmogenesis, Tetralogy of Fallot, and Inter-atrial Communication. The last MICCAI challenge on right ventricle segmentation was held in 2012 and included only 48 cases from a single clinical center. As part of the 12th Workshop on Statistical Atlases and Computational Models of the Heart (STACOM 2021), the M&Ms-2 challenge was organized to promote the interest of the research community around right ventricle segmentation in multi-disease, multi-view, and multi-center cardiac MRI. Three hundred sixty CMR cases, including short-axis and long-axis 4-chamber views, were collected from three Spanish hospitals using nine different scanners from three different vendors, and included a diverse set of right and left ventricle pathologies. The solutions provided by the participants show that nnU-Net achieved the best results overall. However, multi-view approaches were able to capture additional information, highlighting the need to integrate multiple cardiac diseases, views, scanners, and acquisition protocols to produce reliable automatic cardiac segmentation algorithms.

One of the limiting factors for the development and adoption of novel deep-learning (DL) based medical image analysis methods is the scarcity of labeled medical images. Medical image simulation and synthesis can provide solutions by generating ample training data with corresponding ground truth labels. Despite recent advances, generated images demonstrate limited realism and diversity. In this work, we develop a flexible framework for simulating cardiac magnetic resonance (MR) images with variable anatomical and imaging characteristics for the purpose of creating a diversified virtual population. We advance previous works on both cardiac MR image simulation and anatomical modeling to increase the realism in terms of both image appearance and underlying anatomy. To diversify the generated images, we define parameters: 1)to alter the anatomy, 2) to assign MR tissue properties to various tissue types, and 3) to manipulate the image contrast via acquisition parameters. The proposed framework is optimized to generate a substantial number of cardiac MR images with ground truth labels suitable for downstream supervised tasks. A database of virtual subjects is simulated and its usefulness for aiding a DL segmentation method is evaluated. Our experiments show that training completely with simulated images can perform comparable with a model trained with real images for heart cavity segmentation in mid-ventricular slices. Moreover, such data can be used in addition to classical augmentation for boosting the performance when training data is limited, particularly by increasing the contrast and anatomical variation, leading to better regularization and generalization. The database is publicly available at https://osf.io/bkzhm/ and the simulation code will be available at https://github.com/sinaamirrajab/CMRI.

Tracer-kinetic models allow for the quantification of kinetic parameters such as blood flow from dynamic contrast-enhanced magnetic resonance (MR) images. Fitting the observed data with multi-compartment exchange models is desirable, as they are physiologically plausible and resolve directly for blood flow and microvascular function. However, the reliability of model fitting is limited by the low signal-to-noise ratio, temporal resolution, and acquisition length. This may result in inaccurate parameter estimates. This study introduces physics-informed neural networks (PINNs) as a means to perform myocardial perfusion MR quantification, which provides a versatile scheme for the inference of kinetic parameters. These neural networks can be trained to fit the observed perfusion MR data while respecting the underlying physical conservation laws described by a multi-compartment exchange model. Here, we provide a framework for the implementation of PINNs in myocardial perfusion MR. The approach is validated both in silico and in vivo. In the in silico study, an overall decrease in mean-squared error with the ground-truth parameters was observed compared to a standard non-linear least squares fitting approach. The in vivo study demonstrates that the method produces parameter values comparable to those previously found in literature, as well as providing parameter maps which match the clinical diagnosis of patients.

Myocardial blood flow can be quantified from dynamic contrast-enhanced magnetic resonance (MR) images through the fitting of tracer-kinetic models to the observed imaging data. The use of multi-compartment exchange models is desirable as they are physiologically motivated and resolve directly for both blood flow and microvascular function. However, the parameter estimates obtained with such models can be unreliable. This is due to the complexity of the models relative to the observed data which is limited by the low signal-to-noise ratio, the temporal resolution, the length of the acquisitions and other complex imaging artefacts. In this work, a Bayesian inference scheme is proposed which allows the reliable estimation of the parameters of the two-compartment exchange model from myocardial perfusion MR data. The Bayesian scheme allows the incorporation of prior knowledge on the physiological ranges of the model parameters and facilitates the use of the additional information that neighbouring voxels are likely to have similar kinetic parameter values. Hierarchical priors are used to avoid making a priori assumptions on the health of the patients. We provide both a theoretical introduction to Bayesian inference for tracer-kinetic modelling and specific implementation details for this application. This approach is validated in both in silico and in vivo settings. In silico, there was a significant reduction in mean-squared error with the ground-truth parameters using Bayesian inference as compared to using the standard non-linear least squares fitting. When applied to patient data the Bayesian inference scheme returns parameter values that are in-line with those previously reported in the literature, as well as giving parameter maps that match the independant clinical diagnosis of those patients.

Kinetic parameter values, such as myocardial perfusion, can be quantified from dynamic contrast-enhanced magnetic resonance imaging data using tracer-kinetic modeling. However, respiratory motion affects the accuracy of this process. Motion compensation of the image series is difficult due to the rapid local signal enhancement caused by the passing of the gadolinium-based contrast agent. This contrast enhancement invalidates the assumptions of the (global) cost functions traditionally used in intensity-based registrations. The algorithms are unable to distinguish whether the differences in signal intensity between frames are caused by the spatial motion artifacts or the local contrast enhancement. In order to address this problem, a fully automated motion compensation scheme is proposed, which consists of two stages. The first of which uses robust principal component analysis (PCA) to separate the local signal enhancement from the baseline signal, before a refinement stage which uses the traditional PCA to construct a synthetic reference series that is free from motion but preserves the signal enhancement. Validation is performed on 18 subjects acquired in free-breathing and 5 clinical subjects acquired with a breath-hold. The validation assesses the visual quality, the temporal smoothness of tissue curves, and the clinically relevant quantitative perfusion values. The expert observers score the visual quality increased by a mean of 1.58/5 after motion compensation and improvement over the previously published methods. The proposed motion compensation scheme also leads to the improved quantitative performance of motion compensated free-breathing image series [30% reduction in the coefficient of variation across quantitative perfusion maps and 53% reduction in temporal variations (p < 0.001)].

Abstract In radiotherapy, tumors are irradiated with a high dose, while surrounding healthy tissues are spared. To quantify the probability that a tumor is effectively treated with a given dose, statistical models were built and employed in clinical research. These are called tumor control probability (TCP) models. Recently, TCP models started incorporating additional information from imaging modalities. In this way, patient‐specific properties of tumor tissues are included, improving the radiobiological accuracy of models. Yet, the employed imaging modalities are subject to uncertainties with significant impact on the modeling outcome, while the models are sensitive to a number of parameter assumptions. Currently, uncertainty and parameter sensitivity are not incorporated in the analysis, due to time and resource constraints. To this end, we propose a visual tool that enables clinical researchers working on TCP modeling, to explore the information provided by their models, to discover new knowledge and to confirm or generate hypotheses within their data. Our approach incorporates the following four main components: (1) It supports the exploration of uncertainty and its effect on TCP models; (2) It facilitates parameter sensitivity analysis to common assumptions; (3) It enables the identification of inter‐patient response variability; (4) It allows starting the analysis from the desired treatment outcome, to identify treatment strategies that achieve it. We conducted an evaluation with nine clinical researchers. All participants agreed that the proposed visual tool provides better understanding and new opportunities for the exploration and analysis of TCP modeling.

Parallel Coordinate Plots (PCPs) is one of the most powerful techniques for the visualization of multivariate data. However, for large datasets, the representation suffers from clutter due to overplotting. In this case, discerning the underlying data information and selecting specific interesting patterns can become difficult. We propose a new and simple technique to improve the display of PCPs by emphasizing the underlying data structure. Our Orientation-enhanced Parallel Coordinate Plots (OPCPs) improve pattern and outlier discernibility by visually enhancing parts of each PCP polyline with respect to its slope. This enhancement also allows us to introduce a novel and efficient selection method, the Orientation-enhanced Brushing (O-Brushing). Our solution is particularly useful when multiple patterns are present or when the view on certain patterns is obstructed by noise. We present the results of our approach with several synthetic and real-world datasets. Finally, we conducted a user evaluation, which verifies the advantages of the OPCPs in terms of discernibility of information in complex data. It also confirms that O-Brushing eases the selection of data patterns in PCPs and reduces the amount of necessary user interactions compared to state-of-the-art brushing techniques.

Abstract Tumors are heterogeneous tissues consisting of multiple regions with distinct characteristics. Characterization of these intra‐tumor regions can improve patient diagnosis and enable a better targeted treatment. Ideally, tissue characterization could be performed non‐invasively, using medical imaging data, to derive per voxel a number of features, indicative of tissue properties. However, the high dimensionality and complexity of this imaging‐derived feature space is prohibiting for easy exploration and analysis ‐ especially when clinical researchers require to associate observations from the feature space to other reference data, e.g., features derived from histopathological data. Currently, the exploratory approach used in clinical research consists of juxtaposing these data, visually comparing them and mentally reconstructing their relationships. This is a time consuming and tedious process, from which it is difficult to obtain the required insight. We propose a visual tool for: (1) easy exploration and visual analysis of the feature space of imaging‐derived tissue characteristics and (2) knowledge discovery and hypothesis generation and confirmation, with respect to reference data used in clinical research. We employ, as central view, a 2D embedding of the imaging‐derived features. Multiple linked interactive views provide functionality for the exploration and analysis of the local structure of the feature space, enabling linking to patient anatomy and clinical reference data. We performed an initial evaluation with ten clinical researchers. All participants agreed that, unlike current practice, the proposed visual tool enables them to identify, explore and analyze heterogeneous intra‐tumor regions and particularly, to generate and confirm hypotheses, with respect to clinical reference data.

Better understanding of hemodynamics conceivably leads to improved diagnosis and prognosis of cardiovascular diseases. Therefore, an elaborate analysis of the blood-flow in heart and thoracic arteries is essential. Contemporary MRI techniques enable acquisition of quantitative time-resolved flow information, resulting in 4D velocity fields that capture the blood-flow behavior. Visual exploration of these fields provides comprehensive insight into the unsteady blood-flow behavior, and precedes a quantitative analysis of additional blood-flow parameters. The complete inspection requires accurate segmentation of anatomical structures, encompassing a time-consuming and hard-to-automate process, especially for malformed morphologies. We present a way to avoid the laborious segmentation process in case of qualitative inspection, by introducing an interactive virtual probe. This probe is positioned semi-automatically within the blood-flow field, and serves as a navigational object for visual exploration. The difficult task of determining position and orientation along the view-direction is automated by a fitting approach, aligning the probe with the orientations of the velocity field. The aligned probe provides an interactive seeding basis for various flow visualization approaches. We demonstrate illustration-inspired particles, integral lines and integral surfaces, conveying distinct characteristics of the unsteady blood-flow. Lastly, we present the results of an evaluation with domain experts, valuing the practical use of our probe and flow visualization techniques.

Insight into the dynamics of blood-flow considerably improves the understanding of the complex cardiovascular system and its pathologies. Advances in MRI technology enable acquisition of 4D blood-flow data, providing quantitative blood-flow velocities over time. The currently typical slice-by-slice analysis requires a full mental reconstruction of the unsteady blood-flow field, which is a tedious and highly challenging task, even for skilled physicians. We endeavor to alleviate this task by means of comprehensive visualization and interaction techniques. In this paper we present a framework for pre-clinical cardiovascular research, providing tools to both interactively explore the 4D blood-flow data and depict the essential blood-flow characteristics. The framework encompasses a variety of visualization styles, comprising illustrative techniques as well as improved methods from the established field of flow visualization. Each of the incorporated styles, including exploded planar reformats, flow-direction highlights, and arrow-trails, locally captures the blood-flow dynamics and may be initiated by an interactively probed vessel cross-section. Additionally, we present the results of an evaluation with domain experts, measuring the value of each of the visualization styles and related rendering parameters.

Visually assessing the effect of the coronary artery anatomy on the perfusion of the heart muscle in patients with coronary artery disease remains a challenging task. We explore the feasibility of visualizing this effect on perfusion using a numerical approach. We perform a computational simulation of the way blood is perfused throughout the myocardium purely based on information from a three-dimensional anatomical tomographic scan. The results are subsequently visualized using both three-dimensional visualizations and bull's eye plots, partially inspired by approaches currently common in medical practice. Our approach results in a comprehensive visualization of the coronary anatomy that compares well to visualizations commonly used for other scanning technologies. We demonstrate techniques giving detailed insight in blood supply, coronary territories and feeding coronary arteries of a selected region. We demonstrate the advantages of our approach through visualizations that show information which commonly cannot be directly observed in scanning data, such as a separate visualization of the supply from each coronary artery. We thus show that the results of a computational simulation can be effectively visualized and facilitate visually correlating these results to for example perfusion data.

We present novel, comprehensive visualization techniques for the diagnosis of patients with Coronary Artery Disease using segmented cardiac MRI data. We extent an accepted medical visualization technique called the bull's eye plot by removing discontinuities, preserving the volumetric nature of the left ventricular wall and adding anatomical context. The resulting volumetric bull's eye plot can be used for the assessment of transmurality. We link these visualizations to a 3D view that presents viability information in a detailed anatomical context. We combine multiple MRI scans (whole heart anatomical data, late enhancement data) and multiple segmentations (polygonal heart model, late enhancement contours, coronary artery tree). By selectively combining different rendering techniques we obtain comprehensive yet intuitive visualizations of the various data sources.

We have developed a method for automatic contour propagation in cine cardiac magnetic resonance images. The method consists of a new active contour model that tries to maintain a constant contour environment by matching gray values in profiles perpendicular to the contour. Consequently, the contours should maintain a constant position with respect to neighboring anatomical structures, such that the resulting contours reflect the preferences of the user. This is particularly important in cine cardiac magnetic resonance images because local image features do not describe the desired contours near the papillary muscle. The accuracy of the propagation result is influenced by several parameters. Because the optimal setting of these parameters is application dependent, we describe how to use full factorial experiments to optimize the parameter setting. We have applied our method to cine cardiac magnetic resonance image sequences from the long axis two-chamber view, the long axis four-chamber view, and the short axis view. We performed our optimization procedure for each contour in each view. Next, we performed an extensive clinical validation of our method on 69 short axis data sets and 38 long axis data sets. In the optimal parameter setting, our propagation method proved to be fast, robust, and accurate. The resulting cardiac contours are positioned within the interobserver ranges of manual segmentation. Consequently, the resulting contours can be used to accurately determine physiological parameters such as stroke volume and ejection fraction.

This paper presents a new method for deformable model-based segmentation of lumen and thrombus in abdominal aortic aneurysms from computed tomography (CT) angiography (CTA) scans. First the lumen is segmented based on two positions indicated by the user, and subsequently the resulting surface is used to initialize the automated thrombus segmentation method. For the lumen, the image-derived deformation term is based on a simple grey level model (two thresholds). For the more complex problem of thrombus segmentation, a grey level modeling approach with a nonparametric pattern classification technique is used, namely k-nearest neighbors. The intensity profile sampled along the surface normal is used as classification feature. Manual segmentations are used for training the classifier: samples are collected inside, outside, and at the given boundary positions. The deformation is steered by the most likely class corresponding to the intensity profile at each vertex on the surface. A parameter optimization study is conducted, followed by experiments to assess the overall segmentation quality and the robustness of results against variation in user input. Results obtained in a study of 17 patients show that the agreement with respect to manual segmentations is comparable to previous values reported in the literature, with considerable less user interaction.

In recent years, advances in computer technology and a significant increase in the accuracy of medical imaging have made it possible to develop systems that can assist the clinician in diagnosis, planning, and treatment. This paper deals with an area that is generally referred to as computer-assisted surgery, image-directed surgery, or image-guided surgery. We report the research, development, and clinical validation performed since January 1996 in the European Applications in Surgical Interventions (EASI) project, which is funded by the European Commission in their "4th Framework Telematics Applications for Health" program. The goal of this project is the improvement of the effectiveness and quality of image-guided neurosurgery of the brain and image-guided vascular surgery of abdominal aortic aneurysms, while at the same time reducing patient risks and overall cost. We have developed advanced prototype systems for preoperative surgical planning and intraoperative surgical navigation, and we have extensively clinically validated these systems. The prototype systems and the clinical validation results are described in this paper.

The authors focus on image data compression techniques for digital recording. Image coding for storage equipment covers a large variety of systems because the applications differ considerably in nature. Video coding systems suitable for digital TV and HDTV recording and digital electronic still picture storage are considered. In addition, attention is paid to picture coding for interactive systems, such as the compact-disc interactive system. The relation between the recording system boundary conditions and the applied coding techniques is outlined. The main emphasis is on picture coding techniques for digital consumer recording.>

Gradual introduction of HDTV is considered to be important. In this paper, a bit-rate reduction system is introduced, which decreases the bit rate of digital HDTV from 664 Mbit/s to about 80 Mbit/s, while ensuring compatibility with TV. The system is based on first subband splitting the interlaced HDTV signal into an interlaced compatible TV signal and three surplus signals. Then, the compatible TV signal is coded with intraframe DCT coding, whereas the surplus signals are coded with quantization, variable-length coding and runlength coding.

A subband coding system for high quality digital audio signals is described. To achieve low bit rates at a high quality level, it exploits the simultaneous masking effect of the human ear. It is shown how this effect can be used in an adaptive bit-allocation scheme. The proposed approach has been applied in two coding systems, a complex system in which signal is split into 26 subbands, each approximately one third of an octave wide, and a simpler 20-band system. Both systems have been designed for coding stereophonic 16-bit compact disk signals with a sampling frequency of 44.1 kHz. With the 26-band system high-quality results can be obtained at bit rates of 220 kb/s. With the 20-band system, similar results can be obtained at bit rates of 360 kb/s.>

An image coding technique is described that compresses monochrome digital TV images from 8 to about 1 bit/pixel while maintaining high quality. First, for subblocks of 8*8 pixels the discrete cosine transform is calculated. Then, the resulting block of 8*8 transform coefficients is divided into a number of subvectors, each of which is normalized and quantized using vector quantization. The subvector construction and the vector quantization are performed adaptively to the 'direction' of spatial activity in the pixel subblock. The quantization also adapts to the energy of the subvector.>