Surgical instrument segmentation (SIS) is pivotal for robotic-assisted minimally invasive surgery, assisting surgeons by identifying surgical instruments in endoscopic video frames. Recent unsupervised surgical instrument segmentation (USIS) methods primarily rely on pseudo-labels derived from low-level features such as color and optical flow, but these methods show limited effective-ness and generalizability in complex and unseen endo-scopic scenarios. In this work, we propose a label-free unsupervised model featuring a novel module named Multi-View Normalized Cutter (m-NCutter). Different from previous USIS works, our model is trained using a graph-cutting loss function that leverages patch affini-ties for supervision, eliminating the need for pseudo-labels. The framework adaptively determines which affini-ties from which levels should be prioritized. Therefore, the low- and high-level features and their affinities are effectively integrated to train a label-free unsupervised model, showing superior effectiveness and generalization abil-ity. We conduct comprehensive experiments across mul-tiple SIS datasets to validate our approach's state-of-the-art (SOTA) performance, robustness, and exceptional potential as a pre-trained model. Our code is released at https://github.com/MingyuShengSMYIAMNCutter.

Computer-assisted preoperative planning of pelvic fracture reduction surgery has the potential to increase the accuracy of the surgery and to reduce complications. However, the diversity of the pelvic fractures and the disturbance of small fracture fragments present a great challenge to perform reliable automatic preoperative planning. In this paper, we present a comprehensive and automatic preoperative planning pipeline for pelvic fracture surgery. It includes pelvic fracture labeling, reduction planning of the fracture, and customized screw implantation. First, automatic bone fracture labeling is performed based on the separation of the fracture sections. Then, fracture reduction planning is performed based on automatic extraction and pairing of the fracture surfaces. Finally, screw implantation is planned using the adjoint fracture surfaces. The proposed pipeline was tested on different types of pelvic fracture in 14 clinical cases. Our method achieved a translational and rotational accuracy of 2.56 mm and 3.31° in reduction planning. For fixation planning, a clinical acceptance rate of 86.7% was achieved. The results demonstrate the feasibility of the clinical application of our method. Our method has shown accuracy and reliability for complex multi-body bone fractures, which may provide effective clinical preoperative guidance and may improve the accuracy of pelvic fracture reduction surgery.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Informed by modern imaging techniques, current medical navigation systems support physicians during a variety of interventions, such as needle-based operations. During these, an abundance of information is often displayed on monitors placed in positions that are uncomfortable for the operator to view. In this article, we address these issues with the concept and prototype of a customized flexible display that is placed on the patient’s body to provide the essential information at just the right location. We present an empirical evaluation comparing the flexible display against a control condition using a standard interventional monitor setup and an additional condition that combines both. Our results show that the flexible display significantly reduces task load while improving overall usability. Furthermore, we found indications that the flexible display reduces task completion time while also observing a negative effect on accuracy, which needs to be balanced carefully.

Image guidance improves tissue sampling during biopsy by allowing the physician to visualize the tip and trajectory of the biopsy needle relative to the target in MRI, CT, ultrasound, or other relevant imagery. This paper reports a system for fast automatic needle tip and trajectory localization and visualization in MRI that has been developed and tested in the context of an active clinical research program in prostate biopsy. To the best of our knowledge, this is the first reported system for this clinical application and also the first reported system that leverages deep neural networks for segmentation and localization of needles in MRI across biomedical applications. Needle tip and trajectory were annotated on 583 T2-weighted intra-procedural MRI scans acquired after needle insertion for 71 patients who underwent transperineal MRI-targeted biopsy procedure at our institution. The images were divided into two independent training-validation and test sets at the patient level. A deep 3-D fully convolutional neural network model was developed, trained, and deployed on these samples. The accuracy of the proposed method, as tested on previously unseen data, was 2.80-mm average in needle tip detection and 0.98° in needle trajectory angle. An observer study was designed in which independent annotations by a second observer, blinded to the original observer, were compared with the output of the proposed method. The resultant error was comparable to the measured inter-observer concordance, reinforcing the clinical acceptability of the proposed method. The proposed system has the potential for deployment in clinical routine.

Multi-View neuroimaging retrieval and classification play an important role in computer-aided-diagnosis of brain disorders, as multi-view features could provide more insights of the disease pathology and potentially lead to more accurate diagnosis than single-view features. The large inter-feature and inter-subject variations make the multi-view neuroimaging analysis a challenging task. Many multi-view or multi-modal feature fusion methods have been proposed to reduce the impact of inter-feature variations in neuroimaging data. However, there is not much in-depth work focusing on the inter-subject variations. In this study, we propose a subject-centered multi-view feature fusion method for neuroimaging retrieval and classification based on the propagation graph fusion (PGF) algorithm. Two main advantages of the proposed method are: 1) it evaluates the query online and adaptively reshapes the connections between subjects according to the query; 2) it measures the affinity of the query to the subjects using the subject-centered affinity matrices, which can be easily combined and efficiently solved. Evaluated using a public accessible neuroimaging database, our algorithm outperforms the state-of-the-art methods in retrieval and achieves comparable performance in classification.

Medical content-based retrieval (MCBR) plays an important role in computer aided diagnosis and clinical decision support. Multi-modal imaging data have been increasingly used in MCBR, as they could provide more insights of the diseases and complement the deficiencies of single-modal data. However, it is very challenging to fuse data in different modalities since they have different physical fundamentals and large value range variations. In this study, we propose a novel Propagation Graph Fusion (PGF) framework for multi-modal medical data retrieval. PGF models the subjects' relationships in single modalities using the directed propagation graphs, and then fuses the graphs into a single graph by summing up the edge weights. Our proposed PGF method could reduce the large inter-modality and inter-subject variations, and can be solved efficiently using the PageRank algorithm. We test the proposed method on a public medical database with 331 subjects using features extracted from two imaging modalities, PET and MRI. The preliminary results show that our PGF method could enhance multi-modal retrieval and modestly outperform the state-of-the-art single-modal and multi-modal retrieval methods.

Deformable image registration is used increasingly in image-guided interventions and other applications. However, validation and characterization of registration performance remain areas that require further study. We propose an analysis methodology for deriving tolerance limits on the initial conditions for deformable registration that reliably lead to a successful registration. This approach results in a concise summary of the probability of registration failure, while accounting for the variability in the test data. The (β, γ) tolerance limit can be interpreted as a value of the input parameter that leads to successful registration outcome in at least 100β% of cases with the 100γ% confidence. The utility of the methodology is illustrated by summarizing the performance of a deformable registration algorithm evaluated in three different experimental setups of increasing complexity. Our examples are based on clinical data collected during MRI-guided prostate biopsy registered using publicly available deformable registration tool. The results indicate that the proposed methodology can be used to generate concise graphical summaries of the experiments, as well as a probabilistic estimate of the registration outcome for a future sample. Its use may facilitate improved objective assessment, comparison and retrospective stress-testing of deformable.

The National Alliance for Medical Image Computing (NA-MIC), is a multi-institutional, interdisciplinary community of researchers, who share the recognition that modern health care demands improved technologies to ease suffering and prolong productive life. Organized under the National Centers for Biomedical Computing 7 years ago, the mission of NA-MIC is to implement a robust and flexible open-source infrastructure for developing and applying advanced imaging technologies across a range of important biomedical research disciplines. A measure of its success, NA-MIC is now applying this technology to diseases that have immense impact on the duration and quality of life: cancer, heart disease, trauma, and degenerative genetic diseases. The targets of this technology range from group comparisons to subject-specific analysis.

A method for automated location of shape differences in diseased anatomical structures via high resolution biomedical atlases annotated with labels from formal ontologies is described. In particular, a high resolution magnetic resonance image of the myocardium of the human left ventricle was segmented and annotated with structural terms from an extracted subset of the Foundational Model of Anatomy ontology. The atlas was registered to the end systole template of a previous study of left ventricular remodeling in cardiomyopathy using a diffeomorphic registration algorithm. The previous study used thresholding and visual inspection to locate a region of statistical significance which distinguished patients with ischemic cardiomyopathy from those with nonischemic cardiomyopathy. Using semantic technologies and the deformed annotated atlas, this location was more precisely found. Although this study used only a cardiac atlas, it provides a proof-of-concept that ontologically labeled biomedical atlases of any anatomical structure can be used to automate location-based inferences.

Extracting anatomical and functional significant structures renders one of the important tasks for both the theoretical study of the medical image analysis, and the clinical and practical community. In the past, much work has been dedicated only to the algorithmic development. Nevertheless, for clinical end users, a well designed algorithm with an interactive software is necessary for an algorithm to be utilized in their daily work. Furthermore, the software would better be open sourced in order to be used and validated by not only the authors but also the entire community. Therefore, the contribution of the present work is twofolds: first, we propose a new robust statistics based conformal metric and the conformal area driven multiple active contour framework, to simultaneously extract multiple targets from MR and CT medical imagery in 3D. Second, an open source graphically interactive 3D segmentation tool based on the aforementioned contour evolution is implemented and is publicly available for end users on multiple platforms. In using this software for the segmentation task, the process is initiated by the user drawn strokes (seeds) in the target region in the image. Then, the local robust statistics are used to describe the object features, and such features are learned adaptively from the seeds under a non-parametric estimation scheme. Subsequently, several active contours evolve simultaneously with their interactions being motivated by the principles of action and reaction-this not only guarantees mutual exclusiveness among the contours, but also no longer relies upon the assumption that the multiple objects fill the entire image domain, which was tacitly or explicitly assumed in many previous works. In doing so, the contours interact and converge to equilibrium at the desired positions of the desired multiple objects. Furthermore, with the aim of not only validating the algorithm and the software, but also demonstrating how the tool is to be used, we provide the reader reproducible experiments that demonstrate the capability of the proposed segmentation tool on several public available data sets.

Unacceptable execution time of Non-rigid registration (NRR) often presents a major obstacle to its routine clinical use. Parallel computing is an effective way to accelerate NRR. However, development of efficient parallel NRR codes is a very challenging task. One desirable approach is to map the existing sequential algorithm to the parallel architecture to gain speedup instead of designing a new parallel algorithm. Multicores and GPU provide us a cooperative architecture, in which both Single Instruction Multiple Data (SIMD) and Single Program Multiple Data (SPMD) programming models can co-exist and complement each other. We present a method to parallelize a NRR on this cooperative architecture. Our approach is first to separate the sequential algorithm into regular and irregular parts. We then map the regular part on GPU following SIMD paradigm and irregular part on multicores in a SPMD fashion. Unlike the approaches that use multicores or GPU alone, our approach leads to desirable speedup for the whole application by taking advantage of all components of the cooperative parallel architecture, for all individual parts of the application. This helps us to get closer to our goal: cheaper and faster NRR that leads to its more widespread use. The results on clinical brain MRI data show that the GPU-based Block Matching (regular part) can run at least 1.9 times faster than on a typical cluster of workstations with eight high-performance nodes. The multicores-based implementation of the incremental finite element solver (irregular part) achieves speedup of up to 7 times compared to its sequential version. As a result, the total run time of the NRR code can be reduced to less than 1 minute therefore satisfying the real time requirement for its clinical application.

BACKGROUND: A critical challenge in neuroscience is organizing, managing, and accessing the explosion in neuroscientific knowledge, particularly anatomic knowledge. We believe that explicit knowledge-based approaches to make neuroscientific knowledge computationally accessible will be helpful in tackling this challenge and will enable a variety of applications exploiting this knowledge, such as surgical planning. RESULTS: We developed ontology-based models of neuroanatomy to enable symbolic lookup, logical inference and mathematical modeling of neural systems. We built a prototype model of the motor system that integrates descriptive anatomic and qualitative functional neuroanatomical knowledge. In addition to modeling normal neuroanatomy, our approach provides an explicit representation of abnormal neural connectivity in disease states, such as common movement disorders. The ontology-based representation encodes both structural and functional aspects of neuroanatomy. The ontology-based models can be evaluated computationally, enabling development of automated computer reasoning applications. CONCLUSION: Neuroanatomical knowledge can be represented in machine-accessible format using ontologies. Computational neuroanatomical approaches such as described in this work could become a key tool in translational informatics, leading to decision support applications that inform and guide surgical planning and personalized care for neurological disease in the future.

Ultrasound imaging systems provide the clinician with noninvasive, low-cost, and real-time images that can help them in diagnosis, planning, and therapy. However, although the human eye is able to derive the meaningful information from these images, automatic processing is very difficult due to noise and artifacts present in the image. The speckle reducing anisotropic diffusion filter was recently proposed to adapt the anisotropic diffusion filter to the characteristics of the speckle noise present in the ultrasound images and to facilitate automatic processing of images. We analyze the properties of the numerical scheme associated with this filter, using a semi-explicit scheme. We then extend the filter to a matrix anisotropic diffusion, allowing different levels of filtering across the image contours and in the principal curvature directions. We also show a relation between the local directional variance of the image intensity and the local geometry of the image, which can justify the choice of the gradient and the principal curvature directions as a basis for the diffusion matrix. Finally, different filtering techniques are compared on a 2-D synthetic image with two different levels of multiplicative noise and on a 3-D synthetic image of a Y-junction, and the new filter is applied on a 3-D real ultrasound image of the liver.

We introduce an algorithm for segmenting brain magnetic resonance (MR) images into anatomical compartments such as the major tissue classes and neuro-anatomical structures of the gray matter. The algorithm is guided by prior information represented within a tree structure. The tree mirrors the hierarchy of anatomical structures and the subtrees correspond to limited segmentation problems. The solution to each problem is estimated via a conventional classifier. Our algorithm can be adapted to a wide range of segmentation problems by modifying the tree structure or replacing the classifier. We evaluate the performance of our new segmentation approach by revisiting a previously published statistical group comparison between first-episode schizophrenia patients, first-episode affective psychosis patients, and comparison subjects. The original study is based on 50 MR volumes in which an expert identified the brain tissue classes as well as the superior temporal gyrus, amygdala, and hippocampus. We generate analogous segmentations using our new method and repeat the statistical group comparison. The results of our analysis are similar to the original findings, except for one structure (the left superior temporal gyrus) in which a trend-level statistical significance (p = 0.07) was observed instead of statistical significance.

Neurosurgical resection is a therapeutic intervention in the treatment of brain tumors. Precision of the resection can be improved by utilizing Magnetic Resonance Imaging (MRI) as an aid in decision making during Image Guided Neurosurgery (IGNS). Image registration adjusts pre-operative data according to intra-operative tissue deformation. Some of the approaches increase the registration accuracy by tracking image landmarks through the whole brain volume. High computational cost used to render these techniques inappropriate for clinical applications. In this paper we present a parallel implementation of a state of the art registration method, and a number of needed incremental improvements. Overall, we reduced the response time for registration of an average dataset from about an hour and for some cases more than an hour to less than seven minutes, which is within the time constraints imposed by neurosurgeons. For the first time in clinical practice we demonstrated, that with the help of distributed computing non-rigid MRI registration based on volume tracking can be computed intra-operatively.

Ultrasound images provide the clinician with non-invasive, low cost, and real-time images that can help them in diagnosis, planning and therapy. However, although the human eye is able to derive the meaningful information from these images, automatic processing is very difficult because of the noise and artefacts present in the image. In this work, we propose to extend the current anisotropic diffusion technique to deal with the speckle noise present in the Ultrasound images. To this end, we use a previously derived model of the noise, and we write the restoration scheme as a energy minimization constrained by the noise model and parameters. This approach leads to a new data attachment term whose optimal weight can be automatically estimated.

We present a new algorithm to register 3-D preoperative magnetic resonance (MR) images to intraoperative MR images of the brain which have undergone brain shift. This algorithm relies on a robust estimation of the deformation from a sparse noisy set of measured displacements. We propose a new framework to compute the displacement field in an iterative process, allowing the solution to gradually move from an approximation formulation (minimizing the sum of a regularization term and a data error term) to an interpolation formulation (least square minimization of the data error term). An outlier rejection step is introduced in this gradual registration process using a weighted least trimmed squares approach, aiming at improving the robustness of the algorithm. We use a patient-specific model discretized with the finite element method in order to ensure a realistic mechanical behavior of the brain tissue. To meet the clinical time constraint, we parallelized the slowest step of the algorithm so that we can perform a full 3-D image registration in 35 s (including the image update time) on a heterogeneous cluster of 15 personal computers. The algorithm has been tested on six cases of brain tumor resection, presenting a brain shift of up to 14 mm. The results show a good ability to recover large displacements, and a limited decrease of accuracy near the tumor resection cavity.

The next generation of medical systems will integrate multimodality and multimedia data from a variety of sources to aid individual ysicians in providing the best treatment and care for their patients and to help researchers understand patterns of disease in collections of large databases. We promote a principle of maximum flexibility for the design of these systems to accommodate new developments in medical imaging and bioinformatics. We present several characteristics of flexible systems that support medical multimedia. We illustrate how these characteristics can be applied within diverse medical research and clinical therapy environments through two in-depth case studies. Within these examples, we describe how specific emerging technologies, including the semantic web and publish and subscribe networking, can enhance the flexibility of medical multimedia architectures.

The watershed transform has interesting properties that make it useful for many different image segmentation applications: it is simple and intuitive, can be parallelized, and always produces a complete division of the image. However, when applied to medical image analysis, it has important drawbacks (oversegmentation, sensitivity to noise, poor detection of thin or low signal to noise ratio structures). We present an improvement to the watershed transform that enables the introduction of prior information in its calculation. We propose to introduce this information via the use of a previous probability calculation. Furthermore, we introduce a method to combine the watershed transform and atlas registration, through the use of markers. We have applied our new algorithm to two challenging applications: knee cartilage and gray matter/white matter segmentation in MR images. Numerical validation of the results is provided, demonstrating the strength of the algorithm for medical image segmentation.

We describe an animated electro-holographic visualization of brain lesions due to the progression of multiple sclerosis. A research case study is used which documents the expression of visible brain lesions in a series of magnetic resonance imaging (MRI) volumes collected over the interval of one year. Some of the salient information resident within this data is described, and the motivation for using a dynamic spatial display to explore its spatial and temporal characteristics is stated. We provide a brief overview of spatial displays in medical imaging applications, and then describe our experimental visualization pipeline, from the processing of MRI datasets, through model construction, computer graphic rendering, and hologram encoding. The utility, strengths and shortcomings of the electro-holographic visualization are described and future improvements are suggested.

We present a new algorithm for the nonrigid registration of three-dimensional magnetic resonance (MR) intraoperative image sequences showing brain shift. The algorithm tracks key surfaces of objects (cortical surface and the lateral ventricles) in the image sequence using a deformable surface matching algorithm. The volumetric deformation field of the objects is then inferred from the displacements at the boundary surfaces using a linear elastic biomechanical finite-element model. Two experiments on synthetic image sequences are presented, as well as an initial experiment on intraoperative MR images showing brain shift. The results of the registration algorithm show a good correlation of the internal brain structures after deformation, and a good capability of measuring surface as well as subsurface shift. We measured distances between landmarks in the deformed initial image and the corresponding landmarks in the target scan. Cortical surface shifts of up to 10 mm and subsurface shifts of up to 6 mm were recovered with an accuracy of 1 mm or less and 3 mm or less respectively.

Curve evolution schemes for segmentation, implemented with level set methods, have become an important approach in computer vision. Previous work has modeled evolving contours which are curves in 2D or surfaces in 3D. Our objective is to explore recent mathematical work enabling the evolution of manifolds of higher co-dimension. We consider 1D curves in 3D (codimension-two) for the application of automatically, segmenting blood vessels in volumetric magnetic resonance angiography (MRA) images. This paper describes the theoretical foundations of our system, CURVES, then provides segmentation results compared against segmentation obtained interactively by a neurosurgeon. Segmentation of bronchi in lung computed tomography (CT) scans are also presented. The new experiments, comparisons to manual segmentation, and sample comparison to the use of a codimension-one regularization force are the primary contributions of this report.

A magnetic resonance (MR) compatible surgical assistance robot system, designed to aid minimally invasive surgical techniques, such as positioning and directing a catheter or a laser pointer, has been built and is under preclinical evaluation. Two specific issues arise in the MR environment: (i) the reduced space in which to place the robot; and (ii) the MR compatibility. The main mechanical body is located above the head of the surgeon, with two rigid, hanging arms that reach into the workspace. This novel configuration contributes to a small occupancy in the workspace and to the MR compatibility. The robot is also carefully designed for safety and sterilization issues. Details of the kinematics and the design of the robot are given, and the MR compatibility is examined. This shows that the robot has no adverse effect on the imaging, even when it is in motion.

We aimed to study the performance of a parallel implementation of an intraoperative nonrigid registration algorithm that accurately simulates the biomechanical properties of the brain and its deformations during surgery. The algorithm was designed to allow for improved surgical navigation and quantitative monitoring of treatment progress in order to improve the surgical outcome and to reduce the time required in the operating room. We have applied the algorithm to two neurosurgery cases with promising results. High performance computing is a key enabling technology that allows the biomechanical simulation to be executed quickly enough for the algorithm to be practical. Our parallel implementation was evaluated on a symmetric multi-processor and two clusters and exhibited similar performance characteristics on each. The implementation was sufficiently fast to be used in the operating room during a neurosurgery procedure. It allowed a three-dimensional volumetric deformation to be simulated in less than ten seconds.

In this paper, we consider a novel three-dimensional (3-D) visualization technique based on surface flattening for virtual colonoscopy. Such visualization methods could be important in virtual colonoscopy because they have the potential for noninvasively determining the presence of polyps and other pathologies. Further, we demonstrate a method that presents a surface scan of the entire colon as a cine, and affords the viewer the opportunity to examine each point on the surface without distortion. We use certain angle-preserving mappings from differential geometry to derive an explicit method for flattening surfaces obtained from 3-D colon computed tomography (CT) imagery. Indeed, we describe a general method based on a discretization of the Laplace-Beltrami operator for flattening a surface into the plane in a conformal manner. From a triangulated surface representation of the colon, we indicate how the procedure may be implemented using a finite element technique, which takes into account special boundary conditions. We also provide simple formulas that may be used in a real-time cine to correct for distortion.

We give an explicit method for mapping any simply connected surface onto the sphere in a manner which preserves angles. This technique relies on certain conformal mappings from differential geometry. Our method provides a new way to automatically assign texture coordinates to complex undulating surfaces. We demonstrate a finite element method that can be used to apply our mapping technique to a triangulated geometric description of a surface.

This paper describes a novel approach to tissue classification using three-dimensional (3D) derivative features in the volume rendering pipeline. In conventional tissue classification for a scalar volume, tissues of interest are characterized by an opacity transfer function defined as a one-dimensional (1D) function of the original volume intensity. To overcome the limitations inherent in conventional 1D opacity functions, we propose a tissue classification method that employs a multidimensional opacity function, which is a function of the 3D derivative features calculated from a scalar volume as well as the volume intensity. Tissues of interest are characterized by explicitly defined classification rules based on 3D filter responses highlighting local structures, such as edge, sheet, line, and blob, which typically correspond to tissue boundaries, cortices, vessels, and nodules, respectively, in medical volume data. The 3D local structure filters are formulated using the gradient vector and Hessian matrix of the volume intensity function combined with isotropic Gaussian blurring. These filter responses and the original intensity define a multidimensional feature space in which multichannel tissue classification strategies are designed. The usefulness of the proposed method is demonstrated by comparisons with conventional single-channel classification using both synthesized data and clinical data acquired with CT (computed tomography) and MRI (magnetic resonance imaging) scanners. The improvement in image quality obtained using multichannel classification is confirmed by evaluating the contrast and contrast-to-noise ratio in the resultant volume-rendered images with variable opacity values.

In this paper, using certain conformal mappings from uniformization theory, we give an explicit method for flattening the brain surface in a way which preserves angles. From a triangulated surface representation of the cortex, we indicate how the procedure may be implemented using finite elements. Further, we show how the geometry of the brain surface may be studied using this approach.

Image-guided surgery is an application for which high performance computing is increasingly becoming a critical technology. Advances in image-guided surgery techniques have made it possible to acquire images of a patient whilst the surgery is taking place, to align these images with high resolution 3D scans of the patient acquired preoperatively and to merge intraoperative images from multiple imaging modalities. The application of these technologies has now become a routine clinical procedure in some hospitals. However, as the type of procedures undertaken is expanded, it is becoming clear that the use of image fusion and linear registration technology alone has some limitations. We have developed a novel image segmentation algorithm that makes use of an individualized template of normal patient anatomy in order to compute the segmentation of intraoperative imaging data. Intraoperative image segmentation is highly data and compute intensive. In order to achieve accurate segmentation in a time frame compatible with surgical intervention, we have developed a parallel version of our segmentation algorithm, and implemented the algorithm on a symmetric multiprocessor architecture. We have studied the accuracy of the segmentation algorithm, and the scalability and bandwidth requirements of our parallel implementation.

We present a framework for analyzing the shape deformation of structures within the human brain. A mathematical model is developed describing the deformation of any brain structure whose shape is affected by both gross and detailed physical processes. Using our technique, the total shape deformation is decomposed into analytic modes of variation obtained from finite element modeling, and statistical modes of variation obtained from sample data. Our method is general, and can be applied to many problems where the goal is to separate out important from unimportant shape variation across a class of objects. In this paper, we focus on the analysis of diseases that affect the shape of brain structures. Because the shape of these structures is affected not only by pathology but also by overall brain shape, disease discrimination is difficult. By modeling the brain's elastic properties, we are able to compensate for some of the nonpathological modes of shape variation. This allows us to experimentally characterize modes of variation that are indicative of disease processes. We apply our technique to magnetic resonance images of the brains of individuals with schizophrenia, Alzheimer's disease, and normal-pressure hydrocephalus, as well as to healthy volunteers. Classification results are presented.

In this paper we focus on using local 3D structure for segmentation. A tensor descriptor is estimated for each neighbourhood, i.e. for each voxel in the data set. The tensors are created from a combination of the outputs form a set of 3D quadrature filters. The shape of the tensors describe locally the structure of the neighbourhood in terms of how much it is like a plane, a line, and a sphere. We apply this to segmentation of bone from Computer Tomography data (CT). Traditional methods are based purely on gray-level value discrimination and have difficulties in recovering thin bone structures due to so called partial voluming, a problem which is present in all such sampled data. We illuminate the partial voluming problem by showing that thresholding creates complicated artifacts even if the signal is densely enough sampled and can be perfectly reconstructed. The unwanted effects of thresholding can be reduced by a change of the signal basis. We show that by using additional local structure information can significantly reduce the degree of sampling artifacts. Evaluation of the method on a clinical case is presented, the segmentation of a human skull from a CT volume. The method shows that many of the thin bone structures which disappear in a pure thresholding can be recovered.

While the role and utility of Magnetic Resonance Images as a diagnostic tool are well established in current clinical practice, there are a number of emerging medical arenas in which MRI can play an equally important role. In this article, we consider the problem of image-guided surgery, and provide an overview of a series of techniques that we have recently developed in order to automatically utilize MRI-based anatomical reconstructions for surgical guidance and navigation.

We describe a fully-automatic three-dimensional (3-D)-segmentation technique for brain magnetic resonance (MR) images. By means of Markov random fields (MRF's) the segmentation algorithm captures three features that are of special importance for MR images, i.e., nonparametric distributions of tissue intensities, neighborhood correlations, and signal inhomogeneities. Detailed simulations and real MR images demonstrate the performance of the segmentation algorithm. In particular, the impact of noise, inhomogeneity, smoothing, and structure thickness are analyzed quantitatively. Even single-echo MR images are well classified into gray matter, white matter, cerebrospinal fluid, scalp-bone, and background. A simulated annealing and an iterated conditional modes implementation are presented.

There is a need for frameless guidance systems to help surgeons plan the exact location for incisions, to define the margins of tumors, and to precisely identify locations of neighboring critical structures. The authors have developed an automatic technique for registering clinical data, such as segmented magnetic resonance imaging (MRI) or computed tomography (CT) reconstructions, with any view of the patient on the operating table. The authors demonstrate on the specific example of neurosurgery. The method enables a visual mix of live video of the patient and the segmented three-dimensional (3-D) MRI or CT model. This supports enhanced reality techniques for planning and guiding neurosurgical procedures and allows us to interactively view extracranial or intracranial structures nonintrusively. Extensions of the method include image guided biopsies, focused therapeutic procedures, and clinical studies involving change detection over time sequences of images.

Intensity-based classification of MR images has proven problematic, even when advanced techniques are used. Intrascan and interscan intensity inhomogeneities are a common source of difficulty. While reported methods have had some success in correcting intrascan inhomogeneities, such methods require supervision for the individual scan. This paper describes a new method called adaptive segmentation that uses knowledge of tissue intensity properties and intensity inhomogeneities to correct and segment MR images. Use of the expectation-maximization (EM) algorithm leads to a method that allows for more accurate segmentation of tissue types as well as better visualization of magnetic resonance imaging (MRI) data, that has proven to be effective in a study that includes more than 1000 brain scans. Implementation and results are described for segmenting the brain in the following types of images: axial (dual-echo spin-echo), coronal [three dimensional Fourier transform (3-DFT) gradient-echo T1-weighted] all using a conventional head coil, and a sagittal section acquired using a surface coil. The accuracy of adaptive segmentation was found to be comparable with manual segmentation, and closer to manual segmentation than supervised multivariant classification while segmenting gray and white matter.

We developed a three-dimensional (3D) digitized atlas of the human brain to visualize spatially complex structures. It was designed for use with magnetic resonance (MR) imaging data sets. Thus far, we have used this atlas for surgical planning, model-driven segmentation, and teaching. We used a combination of automated and supervised segmentation methods to define regions of interest based on neuroanatomical knowledge. We also used 3D surface rendering techniques to create a brain atlas that would allow us to visualize complex 3D brain structures. We further linked this Information to script files in order to preserve both spatial information and neuroanatomical knowledge. We present here the application of the atlas for visualization in surgical planning far model-driven segmentation and for the teaching of neuroanatomy. This digitized human brain has the potential to provide important reference information for the planning of surgical procedures. It can also serve as a powerful teaching tool, since spatial relationships among neuroanatomical structures can be more readily envisioned when the user is able to view and rotate the structures in 3D space. Moreover, each element of the brain atlas is associated with a name tag, displayed by a user controlled pointer. The atlas holds a major promise as a template for model-driven segmentation. Using this technique, many regions of interest can be characterized simultaneously on new brain images.

There is a need for frameless guidance systems to help neurosurgeons to plan the exact location of a craniotomy, to define the margins of tumors and to precisely identify locations of neighboring critical structures. We have developed an automatic technique for registering clinical data, such as segmented MRI or CT reconstructions, with the patient's head on the operating table. A second method calibrates the position of a video camera relative to the patient. The combination allows a visual mix of live video of the patient with the segmented 3D MRI or CT model, enabling enhanced reality techniques for planning and guiding neurosurgical procedures, and to interactively view extracranial or intracranial structures non-intrusively. Extensions of the method include image guided biopsies, focused therapeutic procedures and clinical studies involving change detection over time sequences of images.>

We present a framework for analyzing the shape of structures within the human brain. A mathematical model is developed describing the deformation of any brain structure whose shape is affected by both gross and detailed physical processes. The total shape deformation is decomposed into physical modes of variation obtained from finite element analysis, and experimental modes of variation obtained from sample data using principal component analysis. This mathematical model is used to classify diseases that affect the shape of the ventricular system of the brain. Because ventricular shape is affected not only by pathology but also by overall brain shape, disease discrimination is difficult. By modeling the brain's elastic properties, we are able to compensate for some of the nonpathological modes of ventricular shape variation. This allows us to experimentally characterize modes of variation that are indicative of disease processes.>

Three dimensional computer models of the anatomy generated from volume acquisitions of computed tomography and magnetic resonance imaging are useful adjuncts to 2D images. This paper describes a system that merges the computer generated 3D models with live video to enhance the surgeon's understanding of the anatomy beneath the surface. The system can be used as a planning aid before the operation and provide additional information during an operation. The application of the system to a brain operation is described.>

In contrast to acquisition-based noise reduction methods a postprocess based on anisotropic diffusion is proposed. Extensions of this technique support 3-D and multiecho magnetic resonance imaging (MRI), incorporating higher spatial and spectral dimensions. The procedure overcomes the major drawbacks of conventional filter methods, namely the blurring of object boundaries and the suppression of fine structural details. The simplicity of the filter algorithm permits an efficient implementation, even on small workstations. The efficient noise reduction and sharpening of object boundaries are demonstrated by applying this image processing technique to 2-D and 3-D spin echo and gradient echo MR data. The potential advantages for MRI, diagnosis, and computerized analysis are discussed in detail.

Hierarchical image segmentation techniques have been applied to spin-echo magnetic resonance images of the brain. From T1- and T2-weighted clinical images, areas representing gray matter, white matter and cerebrospinal fluid were segmented. In the segmentations à priori anatomic information was used and the procedures were performed in an ordered sequence of anatomy.