The development of effective algorithms for removing surgical smoke in laparoscopic surgery has been hindered by the absence of a paired dataset containing real smoky and smoke-free surgical scenes. As a result, existing de-smoking methods have been primarily based on synthetic datasets and non-reference image enhancement metrics, which fail to fully capture the complexity of in vivo surgical scenes. To address this gap, we present a novel paired dataset derived from laparoscopic surgical recordings by identifying video sequences with relatively stationary scenes where smoke emerges. Our approach includes a robust motion-tracking technique that compensates for involuntary patient movements, ensuring reliable pairing of smoky images and their corresponding smoke-free ground truths. From 132 laparoscopic prostatectomy recordings, we curated 41 video sequences, resulting in a dataset of 2000 smoky-to-smoke-free image pairs. From 45 cholecystectomy recordings, we extracted 68 video sequences, resulting in an additional dataset of 1000 image pairs. Using this unique dataset, we evaluated a representative selection of current de-smoking methods, confirming their effectiveness while also highlighting their limitations. Furthermore, we critically revisited the commonly used atmospheric scattering model, atmospheric colour assumptions, and the dark channel prior. Our analysis demonstrated that the traditional atmospheric scattering model with "gray smoke" assumption introduces significant residual errors in the green and blue channels, while the dark channel prior maintains a strong correlation with smoke intensity. These observations suggest that, while less effective for direct smoke separation, the dark channel prior has potential to serve as a useful attention map for deep learning-based de-smoking approaches.

Stereo matching has become an active area of research in the field of computer vision. In minimally invasive surgery, stereo matching provides depth information to surgeons, with the potential to increase the safety of surgical procedures, particularly those performed laparoscopically. Many stereo matching methods have been reported to perform well for natural images, but for images acquired during a laparoscopic procedure, they are limited by image characteristics including illumination differences, weak texture content, specular highlights, and occlusions. To overcome these limitations, we propose a robust edge-preserving stereo matching method for laparoscopic images, comprising an efficient sparse-dense feature matching step, left and right image illumination equalization, and refined disparity optimization. We validated the proposed method using both benchmark biological phantoms and surgical stereoscopic data. Experimental results illustrated that, in the presence of heavy illumination differences between image pairs, texture and textureless surfaces, specular highlights and occlusions, our proposed approach consistently obtains a more accurate estimate of the disparity map than state-of-the-art stereo matching methods in terms of robustness and boundary preservation.

Complete tumor coverage by the thermal ablation zone and with a safety margin (5 or 10 mm) is required to achieve the entire tumor eradication in liver tumor ablation procedures. However, 2D ultrasound (US) imaging has limitations in evaluating the tumor coverage by imaging only one or multiple planes, particularly for cases with multiple inserted applicators or irregular tumor shapes. In this paper, we evaluate the intra-procedural tumor coverage using 3D US imaging and investigate whether it can provide clinically needed information. Using data from 14 cases, we employed surface- and volume-based evaluation metrics to provide information on any uncovered tumor region. For cases with incomplete tumor coverage or uneven ablation margin distribution, we also proposed a novel margin uniformity -based approach to provide quantitative applicator adjustment information for optimization of tumor coverage. Both the surface- and volume-based metrics showed that 5 of 14 cases had incomplete tumor coverage according to the estimated ablation zone. After applying our proposed applicator adjustment approach, the simulated results showed that 92.9% (13 of 14) cases achieved 100% tumor coverage and the remaining case can benefit by increasing the ablation time or power. Our proposed method can evaluate the intra-procedural tumor coverage and intuitively provide applicator adjustment information for the physician. Our 3D US-based method is compatible with the constraints of conventional US-guided ablation procedures and can be easily integrated into the clinical workflow.

In situ visualization of laparoscopic ultrasound in both conventional and robot-assisted laparoscopic surgery requires robust and efficient computation of the pose of the laparoscopic ultrasound probe with respect to the laparoscopic camera. Image-based intrinsic methods of computing this relative pose need to overcome challenges due to irregular illumination, partial feature occlusion, and clutter that are unavoidable in practical laparoscopic surgery. In this paper, we propose an accurate image-based method that is robust to partial occlusion of the fiducials and outliers. The method is extended to multi-view imaging model with applications in stereoscopic laparoscopy and robot-assisted surgery. Rather than treating the model-to-image correspondence and pose computation as separate problems, we solve them jointly using the Kalman Filter-based framework that demonstrates video rate running time (~24fps). By keeping the optical tracking measurements as a reference, we demonstrate that the proposed methods result in clinically acceptable tracking accuracy, reaching target registration errors well below 1.5mm on average. In addition, our multi-view tracking method is compared to a conventional stereo triangulation-based pose estimation scheme that commercial optical tracking systems are based on, to experimentally demonstrate its superiority in terms of accuracy. Finally, we qualitatively demonstrate the suitability of our methods for practical laparoscopic applications by conducting a phantom-based experiment.

Endoscopic vision plays a significant role in minimally invasive surgical procedures. The visibility and maintenance of such direct in situ vision is paramount not only for safety by preventing inadvertent injury but also to improve precision and reduce operating time. Unfortunately, the endoscopic vision is unavoidably degraded due to the illumination variations during surgery. This paper aims to restore or augment such degraded visualization and quantitatively evaluate it during robotic surgery. A multiscale bilateral-weighted retinex method is proposed to remove non-uniform and highly directional illumination and enhance surgical vision, while an objective no-reference image visibility assessment method is defined in terms of sharpness, naturalness, and contrast, to quantitatively and objectively evaluate the endoscopic visualization on surgical video sequences. The methods were validated on surgical data, with the experimental results showing that our method outperforms existent retinex approaches. In particular, the combined visibility was improved from 0.81 to 1.06, while three surgeons generally agreed that the results were restored with much better visibility.

Sensitivity to phase deviations in MRI forms the basis of a variety of techniques, including magnetic susceptibility weighted imaging and chemical shift imaging. Current phase processing techniques fall into two families: those which process the complex image data with magnitude and phase coupled, and phase unwrapping-based techniques that first linearize the phase topology across the image. However, issues, such as low signal and the existence of phase poles, can lead both methods to experience error. Cyclic continuous max-flow (CCMF) phase processing uses primal-dual-variational optimization over a cylindrical manifold, which represent the inherent topology of phase images, increasing its robustness to these issues. CCMF represents a third distinct paradigm in phase processing, being the only technique equipped with the inherent topology of phase. CCMF is robust and efficient with at least comparable accuracy as the prior paradigms.

Fogged surgical field visualization that is a common and potentially harmful problem can lead to inappropriate device use and incorrectly targeted tissue and increase surgical risks in endoscopic surgery. This paper aims to remove fog or smoke on endoscopic video sequences to augment and maintain a direct and clear visualization of the operating field. A new visibility-driven fusion defogging framework is proposed for surgical endoscopic video processing. This framework first recovers the visibility and enhances the contrast of hazy images. To address the color infidelity problem introduced by the visibility recovery, the luminances of the recovered and enhanced images are fused in the gradient domain, and the fused luminance is reconstructed by solving the Poisson equation in the frequency domain. The proposed method is evaluated on clinical videos that were collected from prostate cancer surgery. The experimental results demonstrate that the proposed framework defogs endoscopic images more robustly than currently available methods. Additionally, our method also provides an effective way to improve the visual quality of medical or high-dynamic range images.

Stereo correspondence is a traditional but still challenging problem in various computer vision tasks. Although current stereo matching algorithms work well, they are still limited by occlusions, texture less and blurred structures, and particularly illumination differences. By revisiting the cost construction and aggregation step in the stereo correspondence procedure, this paper studies a multiscale retinex aggregation method to achieve accurate dense stereo matching. Our method employs the retinex theory to effectively enhance local contrast and utilize color information to boost the matching cost construction and aggregation. We evaluate our proposed framework on benchmark and surgical stereo data. The experimental results demonstrate that our multiscale retinex aggregation provides a more or comparable accurate dense stereo matching strategy. In particular, our method is robust to heavy illumination differences while giving similar performance to state-of-the-art methods on images with uniform illumination.

The primary purpose of medical image visualization is to improve patient outcomes by facilitating the inspection, analysis, and interpretation of patient data. This is only possible if the users’ perceptual and cognitive limitations are taken into account during every step of design, implementation, and evaluation of interactive displays. Visualization of medical images, if executed effectively and efficiently, can empower physicians to explore patient data rapidly and accurately with minimal cognitive effort. This article describes a specific case study in biomedical visualization system design and evaluation, which is the visualization of MR angiography images for planning arteriovenous malformation (AVM) interventions. The success of an AVM intervention greatly depends on the surgeon gaining a full understanding of the anatomy of the malformation and its surrounding structures. Accordingly, the purpose of this study was to investigate the usability of visualization modalities involving contour enhancement and stereopsis in the identification and localization of vascular structures using objective user studies. Our preliminary results indicate that contour enhancement, particularly when combined with stereopsis, results in improved performance enhancement of the perception of connectivity and relative depth between different structures.

Minimally invasive valvular intervention commonly requires intra-procedural navigation to provide spatial and temporal information of relevant cardiac structures and device components. Recently intra-procedural trans-esophageal echocardiography (TEE) has been exploited for this purpose due to its accessibility, low cost, ease of use, and real-time imaging capacity. However, the position and orientation of tissue targets relative to surgical tools can be challenging to perceive, particularly using 2D imaging planes. In this paper, we propose the use of CT images to provide a high-quality 3D context to enhance ultrasound images through image registration, providing an augmented guidance system with minimal impact on standard clinical workflow. We also describe an approach to generate synthetic 4D CT images through non-rigid registration of available ultrasound. This can be employed to avoid a requirement for higher radiation. Synthetic CT images were validated through direct comparison of synthetic and real multi-phase CT images. Validation of CT and ultrasound image registration was performed for both dynamic and synthetic CT image datasets. Our results demonstrated that the synthetically generated dynamic CT images provide similar anatomical representation for relevant cardiac anatomy relative to real dynamic CT images, and similar high registration accuracy that can be achieved for intra-procedural TEE to this versus real dynamic CT images.

We present a unique simulator-based methodology for assessing both technical and nontechnical (cognitive) skills for surgical trainees while immersed in a complete medical simulation environment. Further, we have included two crisis scenarios which allow for the evaluation of the effect of cognitive strategy selection on the low-level surgical skills. Training these mixed-mode scenarios can thereby be evaluated on our platform, allowing for improved assessment and a stronger foundation for credentialing, with the potential to reduce the occurrence of adverse events in the operating room. Scientific evaluation and validation of our work is conducted together with 19 junior surgeons in order to achieve the following goals: 1) to provide a qualitative measure of usability, 2) to assess vertebroplasty technical performance of the surgeon, and 3) to explore the relationship between mental workload and surgical performance during crisis. Our results indicate that: 1) the surgeons scored the face validity of our modeled simulation environment very highly ( 4.68 ±0.48, using a 5-point Likert scale), 2) surgeon training enabled completion of tasks more quickly, and 3) the introduction of crisis scenarios negatively affected the surgeons' objective performance. Taken together, our results underscore the need to develop realistic simulation environments that prepare young residents to respond to emergent events in the operating room.

Automating the detection and localization of segmental (regional) left ventricle (LV) abnormalities in magnetic resonance imaging (MRI) has recently sparked an impressive research effort, with promising performances and a breadth of techniques. However, despite such an effort, the problem is still acknowledged to be challenging, with much room for improvements in regard to accuracy. Furthermore, most of the existing techniques are labor intensive, requiring delineations of the endo- and/or epi-cardial boundaries in all frames of a cardiac sequence. The purpose of this study is to investigate a real-time machine-learning approach which uses some image features that can be easily computed, but that nevertheless correlate well with the segmental cardiac function. Starting from a minimum user input in only one frame in a subject dataset, we build for all the regional segments and all subsequent frames a set of statistical MRI features based on a measure of similarity between distributions. We demonstrate that, over a cardiac cycle, the statistical features are related to the proportion of blood within each segment. Therefore, they can characterize segmental contraction without the need for delineating the LV boundaries in all the frames. We first seek the optimal direction along which the proposed image features are most descriptive via a linear discriminant analysis. Then, using the results as inputs to a linear support vector machine classifier, we obtain an abnormality assessment of each of the standard cardiac segments in real-time. We report a comprehensive experimental evaluation of the proposed algorithm over 928 cardiac segments obtained from 58 subjects. Compared against ground-truth evaluations by experienced radiologists, the proposed algorithm performed competitively, with an overall classification accuracy of 86.09% and a kappa measure of 0.73.

Object tracking is a key enabling technology in the context of computer-assisted medical interventions. Allowing the continuous localization of medical instruments and patient anatomy, it is a prerequisite for providing instrument guidance to subsurface anatomical structures. The only widely used technique that enables real-time tracking of small objects without line-of-sight restrictions is electromagnetic (EM) tracking. While EM tracking has been the subject of many research efforts, clinical applications have been slow to emerge. The aim of this review paper is therefore to provide insight into the future potential and limitations of EM tracking for medical use. We describe the basic working principles of EM tracking systems, list the main sources of error, and summarize the published studies on tracking accuracy, precision and robustness along with the corresponding validation protocols proposed. State-of-the-art approaches to error compensation are also reviewed in depth. Finally, an overview of the clinical applications addressed with EM tracking is given. Throughout the paper, we report not only on scientific progress, but also provide a review on commercial systems. Given the continuous debate on the applicability of EM tracking in medicine, this paper provides a timely overview of the state-of-the-art in the field.

We propose a novel multi-region image segmentation approach to extract myocardial scar tissue from 3-D whole-heart cardiac late-enhancement magnetic resonance images in an interactive manner. For this purpose, we developed a graphical user interface to initialize a fast max-flow-based segmentation algorithm and segment scar accurately with progressive interaction. We propose a partially-ordered Potts (POP) model to multi-region segmentation to properly encode the known spatial consistency of cardiac regions. Its generalization introduces a custom label/region order constraint to Potts model to multi-region segmentation. The combinatorial optimization problem associated with the proposed POP model is solved by means of convex relaxation, for which a novel multi-level continuous max-flow formulation, i.e., the hierarchical continuous max-flow (HMF) model, is proposed and studied. We demonstrate that the proposed HMF model is dual or equivalent to the convex relaxed POP model and introduces a new and efficient hierarchical continuous max-flow based algorithm by modern convex optimization theory. In practice, the introduced hierarchical continuous max-flow based algorithm can be implemented on the parallel GPU to achieve significant acceleration in numerics. Experiments are performed in 50 whole heart 3-D LE datasets, 35 with left-ventricular and 15 with right-ventricular scar. The experimental results are compared to full-width-at-half-maximum and Signal-threshold to reference-mean methods using manual expert myocardial segmentations and operator variabilities and the effect of user interaction are assessed. The results indicate a substantial reduction in image processing time with robust accuracy for detection of myocardial scar. This is achieved without the need for additional region constraints and using a single optimization procedure, substantially reducing the potential for error.

We propose a navigation system for transcatheter aortic valve implantation that employs a magnetic tracking system (MTS) along with a dynamic aortic model and intra-operative ultrasound (US) images. This work is motivated by the desire of our cardiology and cardiac surgical colleagues to minimize or eliminate the use of radiation in the interventional suite or operating room. The dynamic 3-D aortic model is constructed from a preoperative 4-D computed tomography dataset that is animated in synchrony with the real time electrocardiograph input of patient, and then preoperative planning is performed to determine the target position of the aortic valve prosthesis. The contours of the aortic root are extracted automatically from short axis US images in real-time for registering the 2-D intra-operative US image to the preoperative dynamic aortic model. The augmented MTS guides the interventionist during positioning and deployment of the aortic valve prosthesis to the target. The results of the aortic root segmentation algorithm demonstrate an error of 0.92±0.85 mm with a computational time of 36.13±6.26 ms. The navigation approach was validated in porcine studies, yielding fiducial localization errors, target registration errors, deployment distance, and tilting errors of 3.02±0.39 mm, 3.31±1.55 mm, 3.23±0.94 mm, and 5.85±3.06(°) , respectively.

In minimally invasive image-guided interventions, different imaging modalities, such as magnetic resonance imaging (MRI), computed tomography (CT), and 3-D ultrasound (US), can provide complementary, multispectral image information. Dynamic image registration is a well-established approach that permits real-time diagnostic information to be enhanced by placing lower-quality real-time images within a high quality anatomical context. For the guidance of cardiac interventions, it would be valuable to register dynamic MRI or CT with intra-operative US. However, in practice, either the high computational cost prohibits such real-time visualization, or else the resulting image quality is not satisfactory for accurate interventional guidance. Modern graphics processing units (GPUs) provide the programmability, parallelism and increased computational precision to address this problem. In this paper, we first outline our research on dynamic 3-D cardiac MR and US image acquisition, real-time dual-modality registration and US tracking. Next, we describe our contributions on image processing and optimization techniques for 4-D (3-D + time) cardiac image rendering, and our GPU-accelerated methodologies for multimodality 4-D medical image visualization and optical blending, along with real-time synchronization of dual-modality dynamic cardiac images. Finally, multiple transfer functions, various image composition schemes, and an extended window-level setting and adjustment approach are proposed and applied to facilitate the dynamic volumetric MR and US cardiac data exploration and enhance the feature of interest of US image that is usually restricted to a narrow voxel intensity range.

This paper outlines a framework for the preoperative planning of robotics-assisted minimally invasive cardiac surgery with application to coronary artery bypass grafting. The intent of the proposed framework is to improve surgical outcomes by considering the intraoperative requirements of the robotic manipulators and the anatomical geometry of the patient's chest. This includes target reachability, instrument dexterity for critical surgical tasks and collision avoidance. Given the patient's preoperative chest computed tomography images, the planning framework aims to determine the optimal location of the access ports on the ribcage, along with the optimal pose of the robotic arms relative to the patient's anatomy. The proposed multi-objective optimality criteria consist of a measure of clearance as well as a new collective kinematic measure. The minimum distances among the robot arms provides a measure for the likelihood of collisions. The proposed kinematic measure is composed of two modified manipulability indices that are dimensionally homogeneous and, in contrast to previously-used measures, are more likely to yield isotropic force and torque distributions when optimized for surgical interventions. The results of a case study illustrate the compatibility of the framework with general guidelines used by experienced surgeons for port selection. Furthermore, the framework surpasses those guidelines by ensuring the feasibility of the solutions in the sense of collision avoidance and surgical target reachability.

In recent years, magnetic tracking systems, whose fundamental unit of measurement is a 5D transformation (three translational and two rotational degrees-of-freedom), have become much more popular. Two 5D sensors can be combined to obtain a 6D transformation similar to the ones provided by the point-based registration in optical tracking. However, estimates of the tool tip uncertainty, which we have called the target tracking error (TTE) since no registration is explicitly performed, are not available in the same manner as their optical counterpart. If the systematic bias error can be corrected and estimates of the 5D or 6D fiducial localizer error (FLE) are provided in the form of zero mean normally distributed random variables in [Formula: see text] and [Formula: see text], respectively, then the TTE can be modeled. In this paper, the required expressions that model the TTE as a function of the systematic bias, FLE and target location are derived and then validated using Monte Carlo simulations. We also show that the first order approximation is sufficient beyond the range of errors typically observed during an image-guided surgery (IGS) procedure. Applications of the models are described for a minimally invasive intracardiac surgical guidance system and needle-based therapy systems. Together with the target registration error (TRE) statistical models for point-based registration, the models presented in this article provide the basic framework for estimating the total system measurement uncertainty for an IGS system. Future work includes developing TRE models for commonly used registration methods that do not already have them.

Two-dimensional ultrasound (US) is widely used in minimally invasive cardiac procedures due to its convenience of use and noninvasive nature. However, the low quality of US images often limits their utility as a means for guiding procedures, since it is often difficult to relate the images to their anatomical context. To improve the interpretability of the US images while maintaining US as a flexible anatomical and functional real-time imaging modality, we describe a multimodality image navigation system that integrates 2D US images with their 3D context by registering them to high quality preoperative models based on magnetic resonance imaging (MRI) or computed tomography (CT) images. The mapping from such a model to the patient is completed using spatial and temporal registrations. Spatial registration is performed by a two-step rapid registration method that first approximately aligns the two images as a starting point to an automatic registration procedure. Temporal alignment is performed with the aid of electrocardiograph (ECG) signals and a latency compensation method. Registration accuracy is measured by calculating the TRE. Results show that the error between the US and preoperative images of a beating heart phantom is 1.7 +/-0.4 mm, with a similar performance being observed in in vivo animal experiments.

Dynamic cardiac magnetic resonance imaging (MR) and computed tomography (CT) provide cardiologists and cardiac surgeons with high-quality 4-D images for diagnosis and therapy, yet the effective use of these high-quality anatomical models remains a challenge. Ultrasound (US) is a flexible imaging tool, but the US images produced are often difficult to interpret unless they are placed within their proper 3-D anatomical context. The ability to correlate real-time 3-D US volumes (RT3D US) with dynamic MR/CT images would offer a significant contribution to improve the quality of cardiac procedures. In this paper, we present a rapid two-step method for registering RT3D US to high-quality dynamic 3-D MR/CT images of the beating heart. This technique overcomes some major limitations of image registration (such as the correct registration result not necessarily occurring at the maximum of the mutual information (MI) metric) using the MI metric. We demonstrate the effectiveness of our method in a dynamic heart phantom (DHP) study and a human subject study. The achieved mean target registration error of CT+US images in the phantom study is 2.59 mm. Validation using human MR/US volumes shows a target registration error of 1.76 mm. We anticipate that this technique will substantially improve the quality of cardiac diagnosis and therapies.

Target registration error (TRE) has become a widely accepted error metric in point-based registration since the error metric was introduced in the 1990s. It is particularly prominent in image-guided surgery (IGS) applications where point-based registration is used in both image registration and optical tracking. In point-based registration, the TRE is a function of the fiducial marker geometry, location of the target and the fiducial localizer error (FLE). While the first two items are easily obtained, the FLE is usually estimated using an a priori technique and applied without any knowledge of real-time information. However, if the FLE can be estimated in real-time, particularly as it pertains to optical tracking, then the TRE can be estimated more robustly. In this paper, a method is presented where the FLE statistics are estimated from the latest measurement of the fiducial registration error (FRE) statistics. The solution is obtained by solving a linear system of equations of the form Ax=b for each marker at each time frame where x are the six independent FLE covariance parameters and b are the six independent estimated FRE covariance parameters. The A matrix is only a function of the tool geometry and hence the inverse of the matrix can be computed a priori and used at each instant in which the FLE estimation is required, hence minimizing the level of computation at each frame. When using a good estimate of the FRE statistics, Monte Carlo simulations demonstrate that the root mean square of the FLE can be computed within a range of 70-90 microm. Robust estimation of the TRE for an optically tracked tool, using a good estimate of the FLE, will provide two enhancements in IGS. First, better patient to image registration will be obtained by using the TRE of the optical tool as a weighting factor of point-based registration used to map the patient to image space. Second, the directionality of the TRE can be relayed back to the surgeon giving the surgeon the option of changing their strategy in order to improve the overall system accuracy and, in turn, the quality of procedure.

Electrophysiological cardiac data mapping is an essential tool for the study of cardiac rhythm disorders, such as atrial fibrillation. Over the past decade, various advanced cardiac mapping systems have been developed to create detailed cardiac maps and assist physicians in diagnosis and therapy guidance. While these systems have increased the ability to study and treat cardiac arrhythmias, inherent limitations exist. The objective of this paper is to describe and evaluate a system that extends current approaches to cardiac mapping, to create a dynamic cardiac map, using patient-specific cardiac models. This paper details novel approaches to collecting a stream of electrophysiological cardiac data, registering the data with patient-specific dynamic cardiac models, and displaying the data directly on the dynamic model surface, giving a more accurate and comprehensive visualization environment when compared to current systems. To validate the system, a series of laboratory and in vivo experiments were conducted. In the laboratory studies, the system was used to test the user's ability to accurately locate a landmark in physical space, as well as their ability to accurately navigate to a virtual location. In the in vivo studies the overall system performance was compared to an existing electrophysiological recording system, where right atrial cardiac maps were created during sinus and paced cardiac rhythms. The results showed that the new dynamic cardiac mapping system was able to maintain high accuracy in locating physical and virtual landmarks, while being able to create a dynamic cardiac map displayed on a dynamic cardiac surface model.

Two-dimensional or 3-D visual guidance is often used for minimally invasive cardiac surgery and diagnosis. This visual guidance suffers from several drawbacks such as limited field of view, loss of signal from time to time, and in some cases, difficulty of interpretation. These limitations become more evident in beating-heart procedures when the surgeon has to perform a surgical procedure in the presence of heart motion. In this paper, we propose dynamic 3-D virtual fixtures (DVFs) to augment the visual guidance system with haptic feedback, to provide the surgeon with more helpful guidance by constraining the surgeon's hand motions thereby protecting sensitive structures. DVFs can be generated from preoperative dynamic magnetic resonance (MR) or computed tomograph (CT) images and then mapped to the patient during surgery. We have validated the feasibility of the proposed method on several simulated surgical tasks using a volunteer's cardiac image dataset. Validation results show that the integration of visual and haptic guidance can permit a user to perform surgical tasks more easily and with reduced error rate. We believe this is the first work presented in the field of virtual fixtures that explicitly considers heart motion.

Error models associated with point-based medical image registration problems were first introduced in the late 1990s. The concepts of fiducial localizer error, fiducial registration error, and target registration error are commonly used in the literature. The model for estimating the target registration error at a position r in a coordinate frame defined by a set of fiducial markers rigidly fixed relative to one another is ubiquitous in the medical imaging literature. The model has also been extended to simulate the target registration error at the point of interest in optically tracked tools. However, the model is limited to describing the error in situations where the fiducial localizer error is assumed to have an isotropic normal distribution in R3. In this work, the model is generalized to include a fiducial localizer error that has an anisotropic normal distribution. Similar to the previous models, the root mean square statistic rms tre is provided along with an extension that provides the covariance Sigma tre. The new model is verified using a Monte Carlo simulation and a set of statistical hypothesis tests. Finally, the differences between the two assumptions, isotropic and anisotropic, are discussed within the context of their use in 1) optical tool tracking simulation and 2) image registration.

The lack of a potential field model capable of providing accurate representations of objects of arbitrary shapes is considered one major limitation in applying the artificial potential field method in many practical applications. In this correspondence, we propose a potential function based on generalized sigmoid functions. The generalized sigmoid model can be constructed from combinations of implicit primitives or from sampled surface data. The constructed potential field model can achieve an accurate analytic description of objects in two or three dimensions and requires very modest computation at run time. In this correspondence, applications of the generalized sigmoid model in path-planning tasks for mobile robots and in haptic feedback tasks are presented. The validation results in this correspondence show that the model can effectively allow the user or mobile robot to avoid penetrations of obstacles while successfully accomplishing the task.

In this paper, we propose a novel multistage method for three-dimensional (3-D) segmentation of medical images and a new radial distance-based segmentation validation approach. For the 3-D segmentation method, we first employ a morphological recursive erosion operation to reduce the connectivity between the region of interest and its surrounding neighborhood; then we design a hybrid segmentation method to achieve an initial result. The hybrid approach integrates an improved fast marching method and a morphological reconstruction algorithm. Finally, a morphological recursive dilation is employed to recover any lost structure from the first stage of the multistage method. This approach is tested on 12 CT and 3 MRI images of the brain, heart, and kidney, to demonstrate the effectiveness and accuracy of this technique across a variety of imaging modalities and organ systems. In order to validate the multistage segmentation method, a novel radial distance-based validation method is proposed that uses a global accuracy (GA) measure. The GA is calculated based on local radial distance errors (LRDE), where LRDE are calculated on the radii emitted from points along the skeleton of the object rather than the centroid, in order to accommodate more complicated organ structures. The experimental results demonstrate that the proposed multistage segmentation method is fast and accurate, with comparable performance to existing segmentation methods, but with a significantly higher execution speed.

Optimization of a similarity metric is an essential component in intensity-based medical image registration. The increasing availability of parallel computers makes parallelizing some registration tasks an attractive option to increase speed. In this paper, two new deterministic, derivative-free, and intrinsically parallel optimization methods are adapted for image registration. DIviding RECTangles (DIRECT) is a global technique for linearly bounded problems, and multidirectional search (MDS) is a recent local method. The performance of DIRECT, MDS, and hybrid methods using a parallel implementation of Powell's method for local refinement, are compared. Experimental results demonstrate that DIRECT and MDS are robust, accurate, and substantially reduce computation time in parallel implementations.

Minimally invasive robotically assisted cardiac surgical systems currently do not routinely employ 3-D image guidance. However, preoperative magnetic resonance and computed tomography (CT) images have the potential to be used in this role, if appropriately registered with the patient anatomy and animated synchronously with the motion of the actual heart. This paper discusses the fusion of optical images of a beating heart phantom obtained from an optically tracked endoscope, with volumetric images of the phantom created from a dynamic CT dataset. High quality preoperative dynamic CT images are created by first extracting the motion parameters of the heart from the series of temporal frames, and then applying this information to animate a high-quality heart image acquired at end systole. Temporal synchronization of the endoscopic and CT model is achieved by selecting the appropriate CT image from the dynamic set, based on an electrocardiographic trigger signal. The spatial error between the optical and virtual images is 1.4 +/- 1.1 mm, while the time discrepancy is typically 50-100 ms. Index Terms-Image guidance, image warping, minimally invasive cardiac surgery, virtual endoscopy, virtual reality.

We present a method of constructing a database of intraoperatively observed human subcortical electrophysiology. In this approach, patient electrophysiological data are standardized using a multiparameter coding system, annotated to their respective magnetic resonance images (MRIs), and nonlinearly registered to a high-resolution MRI reference brain. Once registered, we are able to demonstrate clustering of like interpatient physiologic responses within the thalamus, globus pallidus, subthalamic nucleus, and adjacent structures. These data may in turn be registered to a three-dimensional patient MRI within our image-guided visualization program enabling prior to surgery the delineation of surgical targets, anatomy with high probability of containing specific cell types, and functional borders. The functional data were obtained from 88 patients (106 procedures) via microelectrode recording and electrical stimulation performed during stereotactic neurosurgery at the London Health Sciences Centre. Advantages of this method include the use of nonlinear registration to accommodate for interpatient anatomical variability and the avoidance of digitized versions of printed atlases of anatomy as a common database coordinate system. The resulting database is expandable, easily searched using a graphical user interface, and provides a visual representation of functional organization within the deep brain.

A major limitation of the use of endoscopes in minimally invasive surgery is the lack of relative context between the endoscope and its surroundings. The purpose of this work was to fuse images obtained from a tracked endoscope to surfaces derived from three-dimensional (3-D) preoperative magnetic resonance or computed tomography (CT) data, for assistance in surgical planning, training and guidance. We extracted polygonal surfaces from preoperative CT images of a standard brain phantom and digitized endoscopic video images from a tracked neuro-endoscope. The optical properties of the endoscope were characterized using a simple calibration procedure. Registration of the phantom (physical space) and CT images (preoperative image space) was accomplished using fiducial markers that could be identified both on the phantom and within the images. The endoscopic images were corrected for radial lens distortion and then mapped onto the extracted surfaces via a two-dimensional 2-D to 3-D mapping algorithm. The optical tracker has an accuracy of about 0.3 mm at its centroid, which allows the endoscope tip to be localized to within 1.0 mm. The mapping operation allows multiple endoscopic images to be "painted" onto the 3-D brain surfaces, as they are acquired, in the correct anatomical position. This allows panoramic and stereoscopic visualization, as well as navigation of the 3-D surface, painted with multiple endoscopic views, from arbitrary perspectives.

Magnetic resonance imaging using the echo planar imaging (EPI) technique is particularly sensitive to main (B0) field inhomogeneities. The primary effect is geometrical distortion in the phase encoding direction. In this paper, we present a method based on the conjugate gradient algorithm to correct for this geometrical distortion, by solving the EPI imaging equation. Two versions are presented: one that attempts to solve the full four-dimensional (4-D) imaging equation, and one that independently solves for each profile along the blip encoding direction. Results are presented for both phantom and in vivo brain EPI images and compared with other proposed correction methods.

Many critical functionally distinct subcortical structures are not distinguishable on anatomical magnetic resonance imaging (MRI) scans. In order to provide the neurosurgeon with this missing information, a deformable volumetric atlas of the basal ganglia and thalamus has been created from the Schaltenbrand and Wahren atlas of cryogenic slices. The volumetric atlas can be automatically deformed to an individual patient's MRI. To facilitate the clinical use of the atlas, a visualization platform has been developed for preoperative and intraoperative use which permits manipulation of the merged atlas and MRI data sets in two- and three-dimensional views. The platform includes graphical tools which allow the visualization of projections of a leukotome and other surgical tools with respect to the atlas data, as well as preregistered images from any other imaging modality. In addition, a graphical interface has been designed to create custom virtual lesions using computer models of neurosurgical tools for intraoperative planning. To date this system has been employed as an adjunct to over 30 functional neurosurgical cases including surgery for movement disorders.

For several years the authors have been using a stereoscopic display as a tool in the planning of stereotactic neurosurgical techniques. This PC-based workstation allows the surgeon to interact with and view vascular images in three dimensions, as well as to perform quantitative analysis of the three-dimensional (3-D) space. Some of the perceptual issues relevant to the presentation of medical images on this stereoscopic display were addressed in five experiments. The authors show that a number of parameters-namely the shape, color, and depth cue, associated with a cursor-as well as the image filtering and observer position, have a role in improving the observer's perception of a 3-D image and his ability to localize points within the stereoscopically presented 3-D image. However, an analysis of the results indicates that while varying these parameters can lead to an effect on the performance of individual observers, the effects are not consistent across observers, and the mean accuracy remains relatively constant under the different experimental conditions.

The authors address the use of multimodality imaging as an aid to the planning and guidance of neurosurgical procedures, and discuss the integration of anatomical (CT and MRI), vascular (DSA), and functional (PET) data for presentation to the surgeon during surgery. The authors' workstation is an enhancement of a commercially available system, and in addition to the guidance offered via a hand-held probe, it incorporates the use of multimodality imaging and adds enhanced realism to the surgeon through the use of a stereoscopic three-dimensional (3-D) image display. The probe may be visualized stereoscopically in single or multimodality images. The integration of multimodality data in this manner provides the surgeon with a complete overview of brain structures on which he is performing surgery, or through which he is passing probes or cannulas, enabling him to avoid critical vessels and/or structures of functional significance.