Proton Density Fat Fraction (PDFF) is the gold standard for non-invasive fatty liver diagnosis, but its reliance on Magnetic Resonance Imaging (MRI) limits broad clinical applicability. Motivated by the accessibility of B-mode Ultrasound (US) in fatty liver assessment, we propose a novel framework for liver segmentation and PDFF prediction from US images. To enhance generalization ability despite limited paired US-PDFF data, our framework integrates a cross-task self-supervised pretext task that extracts semantic features to guide echo intensity capture, benefiting both liver segmentation and PDFF prediction. To address the noise and artifacts inherent in US images, our framework leverages T2-weighted imaging (T2WI) exclusively during training to establish a feature bridge between US and PDFF, thereby enhancing PDFF prediction. Once trained, the model relies solely on US for inference, making it a practical and cost-effective alternative to MRI-based PDFF estimation. Additionally, our framework introduces an uncertainty-augmented adversarial loss function to refine liver boundary delineation, further improving segmentation and PDFF prediction accuracy. Experimental results demonstrate that our method outperforms state-of-the-art methods in liver segmentation and PDFF prediction; and in a specific application study, our predicted PDFF achieves accuracy comparable to real PDFF for hepatic steatosis classification, highlighting its clinical potential. The full source code and detailed documentation are publicly available at https://github.com/D0ngZhang/SSTB.

Accurate and efficient surgical robotic tool pose estimation is of fundamental significance to downstream applications such as augmented reality (AR) in surgical training and learning-based autonomous manipulation. While significant advancements have been made in pose estimation for humans and animals, it is still a challenge in surgical robotics due to the scarcity of published data. The relatively large absolute error of the da Vinci end effector kinematics and arduous calibration procedure make calibrated kinematics data collection expensive. Driven by this limitation, we collected a dataset, dubbed SurgPose, providing instance-aware semantic keypoints for visual surgical tool pose estimation and tracking. By marking keypoints using ultraviolet (UV) reactive paint, which is invisible under white light and fluorescent under UV light, we execute the same trajectory under different lighting conditions to collect raw videos and keypoint annotations, respectively. The SurgPose dataset consists of approximately 120 K surgical instrument instances of 6 categories as shown in Fig. 1. Since the videos are collected in stereo pairs, the 2D pose can be lifted to 3D based on stereo-matching depth. In addition to releasing the dataset, we tested a few baseline approaches to surgical instrument tracking to demonstrate the utility of SurgPose. More details can be found at surgpose.github.io.

Tele-ultrasound has the potential greatly to improve health equity for countless remote communities. However, practical scenarios involve potentially large time delays which cause current implementations of telerobotic ultrasound (US) to fail. Using a local model of the remote environment to provide haptics to the expert operator can decrease teleoperation instability, but the delayed visual feedback remains problematic. This paper introduces a robotic tele-US system in which the local model is not only haptic, but also visual, by re-slicing and rendering a pre-acquired US sweep in real time to provide the operator a preview of what the delayed image will resemble. A prototype system is presented and tested with 15 volunteer operators. It is found that visual-haptic model-mediated teleoperation (MMT) compensates completely for time delays up to 1000 ms round trip in terms of operator effort and completion time while conventional MMT does not. Visual-haptic MMT also significantly outperforms MMT for longer time delays in terms of motion accuracy and force control. This proof-of-concept study suggests that visual-haptic MMT may facilitate remote robotic tele-US.

For many applications, remote guidance and telerobotics provide great advantages. For example, tele-ultrasound can bring much-needed expert healthcare to isolated communities. However, existing tele-guidance methods have serious limitations including either low precision for video conference-based systems, or high complexity and cost for telerobotics. A new concept called human teleoperation leverages mixed reality, haptics, and high-speed communication to provide tele-guidance that gives an expert nearly-direct remote control without requiring a robot. This paper provides an overview of the human teleoperation concept and its application to tele-ultrasound. The concept and its impact are discussed. A new approach to remote streaming and control of point-of-care ultrasound systems independent of their manufacturer is described, as is a high-speed communication system for the HoloLens 2 that is compatible with ResearchMode API sensor stream access. Details of these systems are shown in supplementary video demonstrations. Novel interaction methods enabled by HoloLens 2-based pose tracking are also introduced and tests of the communication and user interaction are presented. The results show continued improvement of the system compared to previous work in instrumentation, HCI, and communication. The system thus has good potential for tele-ultrasound, as well as possible other applications of human teleoperation including remote maintenance, inspection, and training. The remote ultrasound streaming and control application is made available open source.

Current teleultrasound methods include audiovisual guidance and robotic teleoperation, which constitute tradeoffs between precision and latency versus flexibility and cost. We present a novel concept of “human teleoperation” which bridges the gap between these two methods. In the concept, an expert remotely teloperates a person (the follower) wearing a mixed-reality headset by controlling a virtual ultrasound probe projected into the person’s scene. The follower matches the pose and force of the virtual device with a real probe. The pose, force, video, ultrasound images, and 3-dimensional mesh of the scene are fed back to the expert. This control framework, where the actuation is carried out by people, allows more precision and speed than verbal guidance, yet is more flexible and inexpensive than robotic teleoperation. The purpose of this paper is to introduce this concept as well as a prototype teleultrasound system with limited haptics and local communication. The system was tested to show its potential, including mean teleoperation latencies of 0.32 ± 0.05 seconds and steady-state errors of 4.4 ± 2.8 mm and 5.4 ± 2.8 ∘ in position and orientation tracking respectively. A preliminary test with an ultrasonographer and four patients was completed, showing lower measurement error and a completion time of 1:36 ± 0:23 minutes using human teleoperation compared to 4:13 ± 3:58 using audiovisual teleguidance.

Graph convolutional neural networks have shown significant potential in natural and histopathology images. However, their use has only been studied in a single magnification or multi-magnification with either homogeneous graphs or only different node types. In order to leverage the multi-magnification information and improve message passing with graph convolutional networks, we handle different embedding spaces at each magnification by introducing the Multi-Scale Relational Graph Convolutional Network (MS-RGCN) as a multiple instance learning method. We model histopathology image patches and their relation with neighboring patches and patches at other scales (i.e., magnifications) as a graph. We define separate message-passing neural networks based on node and edge types to pass the information between different magnification embedding spaces. We experiment on prostate cancer histopathology images to predict the grade groups based on the extracted features from patches. We also compare our MS-RGCN with multiple state-of-the-art methods with evaluations on several source and held-out datasets. Our method outperforms the state-of-the-art on all of the datasets and image types consisting of tissue microarrays, whole-mount slide regions, and whole-slide images. Through an ablation study, we test and show the value of the pertinent design features of the MS-RGCN.

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

This article describes a novel system for quantitative and volumetric measurement of tissue elasticity in the prostate using simultaneous multi-frequency tissue excitation. Elasticity is computed by using a local frequency estimator to measure the three-dimensional local wavelengths of steady-state shear waves within the prostate gland. The shear wave is created using a mechanical voice coil shaker which transmits simultaneous multi-frequency vibrations transperineally. Radio frequency data is streamed directly from a BK Medical 8848 transrectal ultrasound transducer to an external computer where tissue displacement due to the excitation is measured using a speckle tracking algorithm. Bandpass sampling is used that eliminates the need for an ultra-fast frame rate to track the tissue motion and allows for accurate reconstruction at a sampling frequency that is below the Nyquist rate. A roll motor with computer control is used to rotate the transducer and obtain 3D data. Two commercially available phantoms were used to validate both the accuracy of the elasticity measurements as well as the functional feasibility of using the system for in vivo prostate imaging. The phantom measurements were compared with 3D Magnetic Resonance Elastography (MRE), where a high correlation of 96% was achieved. In addition, the system has been used in two separate clinical studies as a method for cancer identification. Qualitative and quantitative results of 11 patients from these clinical studies are presented here. Furthermore, an AUC of 0.87±0.12 was achieved for malignant vs. benign classification using a binary support vector machine classifier trained with data from the latest clinical study with leave one patient out cross-validation.

Tracking of tissue in the surgical environment is often done via locating frame-to-frame keypoint correspondences, and then using these correspondences to warp a prior underlying model such as a spline, mesh, or embedded deformation. We introduce a novel learned model which takes keypoint correspondences as input and enables a prior-free estimation of deformation at any location. For fast point tracking, our model allows for sparse queries, unlike dense grid based CNNs, which run on full images. Our model begins with a novel graph-based point refinement scheme which refines matched keypoints, updating their features and movement instead of discarding possible outliers. Then, we use these refined matches to learn a novel neural implicit representation for estimating movement of any location given its k-nearest neighbor (k-NN) keypoints. We name our implicit deformation model KINFlow (k-NN implicit neural flow). We demonstrate the performance of KINFlow photometrically on three different datasets. KINFlow is the first model to use a graph network to estimate flow of arbitrary query points, and can estimate movement of 1024 points in under 3 ms.

The attention mechanism in artificial neural networks is conceptually interlinked with human visual attention, and studies have shown that either artificial or human attention can facilitate computer vision tasks. However, little research has investigated the visual interpretation of neural network’s attention with the help of human visual attention.

Robot-assisted medical imaging entails the use of a robot to acquire a medical image. Examples include robot-assisted endoscopic camera imaging, ultrasound imaging where the transducer is held by a robot, X-ray imaging where the source and detector are positioned by robots, and actuated capsule endoscopy, where the capsule is maneuvered by external magnetic actuation. Robot assistance enables the controlled trajectory of the imaging system with high precision and accuracy. This makes it possible to compound acquisitions for increased aperture and for volumetric or tomographic imaging, to track medical instrumentation, and to adjust imaging trajectory in a feedback loop as a function of the patient. Intraoperative robotic medical imaging can provide valuable information to the physician and facilitates the registration of preoperative imaging to the patient. In this review article, we describe some of the robotic imaging systems developed for diagnosis and intervention guidance. In particular, based on the surveyed research activity, we will describe our view of the state of the art in ultrasound, endoscopy, X-ray, optical coherence tomography, and nuclear medicine. We will discuss approaches to autonomous scanning and physics-driven approaches such as elastography and photoacoustic tomography, where the accurate placement and trajectory control of the imaging system using a robot are of paramount importance. We will map out the current state of the art and discuss potential avenues of research.

In shear wave absolute vibro-elastography (S-WAVE), a steady-state multi-frequency external mechanical excitation is applied to tissue, while a time-series of ultrasound radio-frequency (RF) data are acquired. Our objective is to determine the potential of S-WAVE to classify breast tissue lesions as malignant or benign. We present a new processing pipeline for feature-based classification of breast cancer using S-WAVE data, and we evaluate it on a new data set collected from 40 patients. Novel bi-spectral and Wigner spectrum features are computed directly from the RF time series and are combined with textural and spectral features from B-mode and elasticity images. The Random Forest permutation importance ranking and the Quadratic Mutual Information methods are used to reduce the number of features from 377 to 20. Support Vector Machines and Random Forest classifiers are used with leave-one-patient-out and Monte Carlo cross-validations. Classification results obtained for different feature sets are presented. Our best results (95% confidence interval, Area Under Curve = 95%±1.45%, sensitivity = 95%, and specificity = 93%) outperform the state-of-the-art reported S-WAVE breast cancer classification performance. The effect of feature selection and the sensitivity of the above classification results to changes in breast lesion contours is also studied. We demonstrate that time-series analysis of externally vibrated tissue as an elastography technique, even if the elasticity is not explicitly computed, has promise and should be pursued with larger patient datasets. Our study proposes novel directions in the field of elasticity imaging for tissue classification.

We introduce two model-based iterative methods to obtain shear modulus images of tissue using magnetic resonance elastography. The first method jointly finds the displacement field that best fits tissue displacement data and the corresponding shear modulus. The displacement satisfies a viscoelastic wave equation constraint, discretized using the finite element method. Sparsifying regularization terms in both shear modulus and displacement are used in the cost function minimized for the best fit. The second method extends the first method for multifrequency tissue displacement data. The formulated problems are bi-convex. Their solution can be obtained iteratively by using the alternating direction method of multipliers. Sparsifying regularizations and the wave equation constraint filter out sensor noise and compressional waves. Our methods do not require bandpass filtering as a preprocessing step and converge fast irrespective of the initialization. We evaluate our new methods in multiple in silico and phantom experiments, with comparisons with existing methods, and we show improvements in contrast to noise and signal-to-noise ratios. Results from an in vivo liver imaging study show elastograms with mean elasticity comparable to other values reported in the literature.

Deep CNN-based methods have so far achieved the state of the art results in multi-view 3D object reconstruction. Despite the considerable progress, the two core modules of these methods - view feature extraction and multi-view fusion, are usually investigated separately, and the relations among multiple input views are rarely explored. Inspired by the recent great success in Transformer models, we reformulate the multi-view 3D reconstruction as a sequence-to-sequence prediction problem and propose a framework named 3D Volume Transformer. Unlike previous CNN-based methods using a separate design, we unify the feature extraction and view fusion in a single Transformer network. A natural advantage of our design lies in the exploration of view-to-view relationships using self-attention among multiple unordered inputs. On ShapeNet - a large-scale 3D reconstruction benchmark, our method achieves a new state-of-the-art accuracy in multi-view reconstruction with fewer parameters (70% less) than CNN-based methods. Experimental results also suggest the strong scaling capability of our method. Our code will be made publicly available.

Quantitative ultrasound (QUS) offers a non-invasive and objective way to quantify tissue health. We recently presented a spatially adaptive regularization method for reconstruction of a single QUS parameter, limited to a two dimensional region. That proof-of-concept study showed that regularization using homogeneity prior improves the fundamental precision-resolution trade-off in QUS estimation. Based on the weighted regularization scheme, we now present a multiparametric 3D weighted QUS (3D QUS) method, involving the reconstruction of three QUS parameters: attenuation coefficient estimate (ACE), integrated backscatter coefficient (IBC) and effective scatterer diameter (ESD). With the phantom studies, we demonstrate that our proposed method accurately reconstructs QUS parameters, resulting in high reconstruction contrast and therefore improved diagnostic utility. Additionally, the proposed method offers the ability to analyze the spatial distribution of QUS parameters in 3D, which allows for superior tissue characterization. We apply a three-dimensional total variation regularization method for the volumetric QUS reconstruction. The 3D regularization involving N planes results in a high QUS estimation precision, with an improvement of standard deviation over the theoretical 1/N rate achievable by compounding N independent realizations. In the in vivo liver study, we demonstrate the advantage of adopting a multiparametric approach over the single parametric counterpart, where a simple quadratic discriminant classifier using feature combination of three QUS parameters was able to attain a perfect classification performance to distinguish between normal and fatty liver cases.

Magnetic resonance elastography (MRE) is commonly regarded as the imaging-based gold-standard for liver fibrosis staging, comparable to biopsy. While ultrasound-based elastography methods for liver fibrosis staging have been developed, they are confined to a 1D or a 2D region of interest and to a limited depth. 3D Shear Wave Absolute Vibro-Elastography (S-WAVE) is a steady-state, external excitation, volumetric elastography technique that is similar to MRE, but has the additional advantage of multi-frequency excitation. We present a novel ultrasound matrix array implementation of S-WAVE that takes advantage of 3D imaging. We use a matrix array transducer to sample axial multi-frequency steady-state tissue motion over a volume, using a Color Power Angiography sequence. Tissue motion with the frequency components (40, 50,60) and (45, 55, 65) Hz are acquired over a (90° lateral)×(40° elevational)×(16 cm depth) sector with an acquisition time of 12 seconds. We compute the elasticity map in 3D using local spatial frequency estimation. We characterize this new approach in tissue phantoms against measurements obtained with transient-elastography and MRE. Six healthy volunteers and eight patients with chronic liver disease were imaged. Their MRE and S-WAVE volumes were aligned using T1 to B-mode registration for direct comparison in common regions of interest. S-WAVE and MRE results are correlated with R2= 0.92, while MRE and TE results are correlated with R2= 0.71. Our findings show that S-WAVE with matrix array has the potential to deliver a similar assessment of liver fibrosis as MRE in a more accessible, inexpensive way, to a broader set of patients.

In this paper, we evaluate the effect of increasing camera baselines on depth perception in robot-assisted surgery. Restricted by the diameter of the surgical trocar through which they are inserted, current clinical stereo endoscopes have a fixed baseline of 5.5 mm. To overcome this restriction, we propose using a stereoscopic "pickup" camera with a side-firing design that allows for larger baselines. We conducted a user study with baselines of 10 mm, 15 mm, 20 mm, and 30 mm to evaluate the effect of increasing baseline on depth perception when used with the da Vinci surgical system. Subjects (N=28) were recruited and asked to rank differently sized poles, mounted at a distance of 200 mm from the cameras, according to their increasing order of height when viewed under different baseline conditions. The results showed that subjects performed better as the baseline was increased with the best performance at a 20 mm baseline. This preliminary proof-of-concept study shows that there is opportunity to improve depth perception in robot-assisted surgical systems with a change in endoscope design philosophy. In this paper, we present this change with our side-firing "pickup" camera and its flexible baseline design. Ultimately, this serves as the first step towards an adaptive baseline camera design that maximizes depth perception in surgery.

Visual inspection of histopathology images of stained biopsy tissue by expert pathologists is the standard method for grading of prostate cancer (PCa). However, this process is time-consuming and subject to high inter-observer variability. Machine learning-based methods have the potential to improve efficient throughput of large volumes of slides while decreasing variability, but they are not easy to develop because they require substantial amounts of labeled training data. In this paper, we propose a deep learning-based classification technique and data augmentation methods for accurate grading of PCa in histopathology images in the presence of limited data. Our method combines the predictions of three separate convolutional neural networks (CNNs) that work with different patch sizes. This enables our method to take advantage of the greater amount of contextual information in larger patches as well as greater quantity of smaller patches in the labeled training data. The predictions produced by the three CNNs are combined using a logistic regression model, which is trained separately after the CNN training. To effectively train our models, we propose new data augmentation methods and empirically study their effects on the classification accuracy. The proposed method achieves an accuracy of [Formula: see text] in classifying cancerous patches versus benign patches and an accuracy of [Formula: see text] in classifying low-grade (i.e., Gleason grade 3) from high-grade (i.e., Gleason grades 4 and 5) patches. The agreement level of our automatic grading method with expert pathologists is within the range of agreement between pathologists. Our experiments indicate that data augmentation is necessary for achieving expert-level performance with deep learning-based methods. A combination of image-space augmentation and feature-space augmentation leads to the best results. Our study shows that well-designed and properly trained deep learning models can achieve PCa Gleason grading accuracy that is comparable to an expert pathologist.

The Hausdorff Distance (HD) is widely used in evaluating medical image segmentation methods. However, the existing segmentation methods do not attempt to reduce HD directly. In this paper, we present novel loss functions for training convolutional neural network (CNN)-based segmentation methods with the goal of reducing HD directly. We propose three methods to estimate HD from the segmentation probability map produced by a CNN. One method makes use of the distance transform of the segmentation boundary. Another method is based on applying morphological erosion on the difference between the true and estimated segmentation maps. The third method works by applying circular/spherical convolution kernels of different radii on the segmentation probability maps. Based on these three methods for estimating HD, we suggest three loss functions that can be used for training to reduce HD. We use these loss functions to train CNNs for segmentation of the prostate, liver, and pancreas in ultrasound, magnetic resonance, and computed tomography images and compare the results with commonly-used loss functions. Our results show that the proposed loss functions can lead to approximately 18-45% reduction in HD without degrading other segmentation performance criteria such as the Dice similarity coefficient. The proposed loss functions can be used for training medical image segmentation methods in order to reduce the large segmentation errors.

PCa is a disease with a wide range of tissue patterns and this adds to its classification difficulty. Moreover, the data source heterogeneity, i.e. inconsistent data collected using different machines, under different conditions, by different operators, from patients of different ethnic groups, etc., further hinders the effectiveness of training a generalized PCa classifier. In this paper, for the first time, a Generative Adversarial Network (GAN)-based three-player minimax game framework is used to tackle data source heterogeneity and to improve PCa classification performance, where a proposed modified U-Net is used as the encoder. Our dataset consists of novel high-frequency ExactVu ultrasound (US) data collected from 693 patients at five data centers. Gleason Scores (GSs) are assigned to the 12 prostatic regions of each patient. Two classification tasks: benign vs. malignant and low- vs. high-grade, are conducted and the classification results of different prostatic regions are compared. For benign vs. malignant classification, the three-player minimax game framework achieves an Area Under the Receiver Operating Characteristic (AUC) of 93.4%, a sensitivity of 95.1% and a specificity of 87.7%, respectively, representing significant improvements of 5.0%, 3.9%, and 6.0% compared to those of using heterogeneous data, which confirms its effectiveness in terms of PCa classification.

This paper evaluates the feasibility of a novel optical sensing concept to measure forces applied at the tip of daVinci EndoWrist instruments. An optical slit is clamped onto the instrument shaft, in-line with an infrared LED-bicell pair. Deflection of the shaft moves the slit with respect to the LED-bicell pair and modulates the light incident on each active element of the bicell. The differential photocurrent is conditioned and monitored to estimate the tip forces. The feasibility evaluation consists of a flexible beam model to quantify the required sensor performance, experimental results with a 3D printed prototype and estimation of the sensor limitations including the measurement bandwidth due to the structural dynamics. The proposed approach requires no modifications to the instrument, is adaptable to different instruments and robot platforms, and leads to high-resolution, high-dynamic range sensing without hysteresis.

Augmented reality (AR) is a promising technology where the surgeon can see the medical abnormality in the context of the patient. It makes the anatomy of interest visible to the surgeon which otherwise is not visible. It can result in better surgical precision and therefore, potentially better surgical outcomes and faster recovery times. Despite these benefits, the current AR systems suffer from two major challenges; first, incorrect depth perception and, second, the lack of suitable evaluation systems. Therefore, in the current paper we addressed both of these problems. We proposed a color depth encoding (CDE) technique to estimate the distance between the tumor and the tissue surface using a surgical instrument. We mapped the distance between the tumor and the tissue surface to the blue-red color spectrum. For evaluation and interaction with our AR technique, we propose to use a virtual surgical instrument method using the CAD model of the instrument. The users were asked to reach the judged distance in the surgical field using the virtual tool. Realistic tool movement was simulated by collecting the forward kinematics joint encoder data. The results showed significant improvement in depth estimation, time for task completion and confidence, using our CDE technique with and without stereo versus other two cases, that are, Stereo-No CDE and No Stereo-No CDE.

Incorrect depth perception and lack of good evaluation systems are major barriers in clinical translation of augmented and mixed reality AR/MR. Thus, a systematic study of depth cues is necessary. Therefore, in the current paper we present a method to introduce the quantitative depth cue Motion Parallax (MP) in surgical scenes and study its effect on depth perception when combined with binocular disparity. In addition to this we present an innovative virtual tool method to evaluate depth. To introduce MP, we reconstructed the tissue surface using structure from motion technique. Then to get accurate absolute scale of the reconstructed surface stereo-triangulation was used. The simulated tumor was rendered beneath the reconstructed point-cloud by rendering a hole for `X-ray' like vision. The MP was introduced by rotating the entire scene from side-to-side with a tumor-surface-point as pivot for maximum impact. Finally for evaluation, we used a virtual surgical tool rendered using real-time da Vinci surgical API's forward kinematics data. In total, 12 subjects participated in a within-subjects-experiment design to study four cases, i.e., Stereo + MP ( S+MP), Mono + MP ( M+MP), Stereo + No MP (S+N-MP) and Mono + No MP (M+N-MP). The subjects significantly overestimated Judged Percentage of True Distance in M+MP when compared to M+N-MP (probability (p)=0.000, Number of Samples (N))=120) and S+N-MP cases ( p=0.001, N = 120). Furthermore, the observed VariableError was less in S+MP and S+N-MP cases when compared to M+MP and M+N-MP cases. The use of Motion Parallax in console interfaces for surgical robotics showed overestimation of judged distance. But to our knowledge it is the first work studying the effect of motion parallax and stereo in the surgical context. Therefore, its further study is warranted.

We compare different possible scanning geometries for prostate photoacoustic tomography (PAT) while considering a realistic reconstruction scenario in which the limited view of the prostate and the directivity effect of the transducer are considered. Simulations and experiments confirm that an intra-operative configuration in which the photoacoustic signal is received by a pickup transducer from the anterior surface of the prostate provides the best approach. We propose a PAT acquisition system that includes a da Vinci system controlled by the da Vinci Research Kit, an illumination laser, and an ultrasound machine with parallel data acquisition. The robot maneuvers the pickup transducer to form a cylindrical detection surface around the prostate. The robot is programmed to acquire trajectories in which the transducer face is parallel to and oriented toward a rotational tomography axis, while the laser is fired and PAT data are collected at regular intervals. We present our initial images acquired with this novel system.

We present a novel technique for real-time deformable registration of 3-D to 2.5-D transrectal ultrasound (TRUS) images for image-guided, robot-assisted laparoscopic radical prostatectomy (RALRP). For RALRP, a pre-operatively acquired 3-D TRUS image is registered to thin-volumes comprised of consecutive intra-operative 2-D TRUS images, where the optimal transformation is found using a gradient descent method based on analytical first and second order derivatives. Our method relies on an efficient algorithm for real-time extraction of arbitrary slices from a 3-D image deformed given a discrete mesh representation. We also propose and demonstrate an evaluation method that generates simulated models and images for RALRP by modeling tissue deformation through patient-specific finite-element models (FEM). We evaluated our method on in-vivo data from 11 patients collected during RALRP and focal therapy interventions. In the presence of an average landmark deformation of 3.89 and 4.62 mm, we achieved accuracies of 1.15 and 0.72 mm, respectively, on the synthetic and in-vivo data sets, with an average registration computation time of 264 ms, using MATLAB on a conventional PC. The results show that the real-time tracking of the prostate motion and deformation is feasible, enabling a real-time augmented reality-based guidance system for RALRP.].

In quantitative elastography, maps of the mechanical properties of soft tissue, or elastograms, are calculated from the measured displacement data by solving an inverse problem. The model assumptions have a significant effect on elastograms. Motivated by the high sensitivity of imaging results to the model assumptions for in vivo magnetic resonance elastography of the prostate, we compared elastograms obtained with four different methods. Two finite-element method (FEM)-based methods developed by our group were compared with two other commonly used methods, local frequency estimator (LFE) and curl-based direct inversion (c-DI). All the methods assume a linear isotropic elastic model, but the methods vary in their assumptions, such as local homogeneity or incompressibility, and in the specific approach used. We report results using simulations, phantom, and ex vivo and in vivo data. The simulation and phantom studies show, for regions with an inclusion, that the contrast to noise ratio (CNR) for the FEM methods is about three to five times higher than the CNR for the LFE and c-DI and the rms error is about half. The LFE method produces very smooth results (i.e., low CNR) and is fast. c-DI is faster than the FEM methods but it is only accurate in areas where elasticity variations are small. The artifacts resulting from the homogeneity assumption in c-DI is detrimental in regions with large variations. The ex vivo and in vivo results also show similar trends as the simulation and phantom studies. The c-FEM method is more sensitive to noise compared with the mixed-FEM due to higher orders derivatives. This is especially evident at lower frequencies, where the wave curvature is smaller and it is more prone to such error, causing a discrepancy in the absolute values between the mixed-FEM and c-FEM in our in vivo results. In general, the proposed FEMs use fewer simplifying assumptions and outperform the other methods but they are computationally more expensive.

This paper describes the implementation of a heart motion compensation system on the da Vinci surgical system (Intuitive Surgical Inc.) with the da Vinci Research Kit, for the purpose of simulating minimally invasive coronary artery bypass surgery on the beating heart. A Novint Falcon device is used to simulate the motion of the coronary bypass site. The 3D position of the heart simulator is measured optically in real time using an NDI Optotrak Certus tracker. The NDI measurements are used to command the da Vinci patient side manipulators to track, from a fixed distance, the simulated heart surface. Visual stabilization is achieved by having a stereo endoscopic camera carried by another patient-side manipulator that is also tracking the heart surface. The system performance is evaluated and discussed. In a preliminary evaluation study, surgeons were asked to perform a bimanual teleoperation suturing task on the simulated heart, with the gold standard being a suturing task on a stationary surface. When motion compensation was enabled, the median completion time dropped from 1.24 to 1.07 of the gold standard, the number of errors was reduced, and subjective measures show higher preference for the use of motion compensation in the da Vinci controllers.

Low-dose-rate prostate brachytherapy treatment takes place by implantation of small radioactive seeds in and sometimes adjacent to the prostate gland. A patient specific target anatomy for seed placement is usually determined by contouring a set of collected transrectal ultrasound images prior to implantation. Standard-of-care in prostate brachytherapy is to delineate the clinical target anatomy, which closely follows the real prostate boundary. Subsequently, the boundary is dilated with respect to the clinical guidelines to determine a planning target volume. Manual contouring of these two anatomical targets is a tedious task with relatively high observer variability. In this work, we aim to reduce the segmentation variability and planning time by proposing an efficient learning-based multi-label segmentation algorithm. We incorporate a sparse representation approach in our methodology to learn a dictionary of sparse joint elements consisting of images, and clinical and planning target volume segmentation. The generated dictionary inherently captures the relationships among elements, which also incorporates the institutional clinical guidelines. The proposed multi-label segmentation method is evaluated on a dataset of 590 brachytherapy treatment records by 5-fold cross validation. We show clinically acceptable instantaneous segmentation results for both target volumes.

This paper describes the implementation of a heart motion compensation system on the da Vinci surgical system (Intuitive Surgical Inc.) for coronary artery bypass surgery. By introducing a robot-assisted solution, this surgery could be performed completely minimally invasively and on a beating heart. In this work we describe the development of open loop controllers based on spectral line decomposition and the assumption of a periodic trajectory. This allows the da Vinci patient-side manipulators to track an actual heart trajectory with sub-millimetre error. Further, to simulate a virtually stabilized environment, we present the novel concept of maintaining the camera fixed relative to the heart target, effectively decoupling the vision tracking and arm tracking challenges. Finally, we executed preliminary experiments to evaluate surgeons' ability to perform simulated suturing and peg transfer tasks on a moving target. Performance for the simulated suturing was evaluated based on task completion time, accuracy of needle placement, and number of errors. For the suture task, the number of missed targets decreased from 37% to 13% when compensation was enabled, the number of hit targets increased from 26% to 41%, and completion time decreased. For the peg transfer tasks, again completion time and number of errors were measured. Though the margin for error was larger, there was less perceived difficulty of the task when compensation was enabled.

The integration of eye-gaze tracking at the console of surgical robots has the potential to add both speed and functionality to the human-robot interface. In this paper, we present a novel eye gaze tracker that can be integrated as a simple retrofit to the da Vinci console. In particular, the eye tracker can be used with the da Vinci Research Kit (dVRK) to control the patient side manipulators. We present the eye-tracker design and calibration. First, a 2D calibration is carried out to estimate the gaze on the da Vinci's stereoscopic display, followed by a 3D “hand-eye” calibration to estimate the gaze in the surgical scene. Using the dVRK, we demonstrated the use of the eye-tracker to perform a gaze-assisted task, in which the estimated gaze is used as a set point for the robot and the user is guided towards the point of gaze through haptic feedback. The task performed was peg placement and was evaluated in a preliminary user study with five users.

Low-dose-rate brachytherapy is a radiation treatment method for localized prostate cancer. The standard of care for this treatment procedure is to acquire transrectal ultrasound images of the prostate in order to devise a plan to deliver sufficient radiation dose to the cancerous tissue. Brachytherapy planning involves delineation of contours in these images, which closely follow the prostate boundary, i.e., clinical target volume. This process is currently performed either manually or semi-automatically, which requires user interaction for landmark initialization. In this paper, we propose a multi-atlas fusion framework to automatically delineate the clinical target volume in ultrasound images. A dataset of a priori segmented ultrasound images, i.e., atlases, is registered to a target image. We introduce a pairwise atlas agreement factor that combines an image-similarity metric and similarity between a priori segmented contours. This factor is used in an atlas selection algorithm to prune the dataset before combining the atlas contours to produce a consensus segmentation. We evaluate the proposed segmentation approach on a set of 280 transrectal prostate volume studies. The proposed method produces segmentation results that are within the range of observer variability when compared to a semi-automatic segmentation technique that is routinely used in our cancer clinic.

This work reports the use of ultrasound radio frequency (RF) time series analysis as a method for ultrasound-based classification of malignant breast lesions. The RF time series method is versatile and requires only a few seconds of raw ultrasound data with no need for additional instrumentation. Using the RF time series features, and a machine learning framework, we have generated malignancy maps, from the estimated cancer likelihood, for decision support in biopsy recommendation. These maps depict the likelihood of malignancy for regions of size 1 mm(2) within the suspicious lesions. We report an area under receiver operating characteristics curve of 0.86 (95% confidence interval [CI]: 0.84%-0.90%) using support vector machines and 0.81 (95% CI: 0.78-0.85) using Random Forests classification algorithms, on 22 subjects with leave-one-subject-out cross-validation. Changing the classification method yielded consistent results which indicates the robustness of this tissue typing method. The findings of this report suggest that ultrasound RF time series, along with the developed machine learning framework, can help in differentiating malignant from benign breast lesions, subsequently reducing the number of unnecessary biopsies after mammography screening.

This paper describes the applicability of an asymmetric force feedback control framework for bimanual robot-assisted surgery using the da Vinci surgical system (Intuitive Surgical Inc.). The core idea of this method, previously presented in [1], is that when completing two-handed tasks involving an action and a reaction force, the forces applied on the environment by the action hand are not transferred back to the same hand, but rather to the reaction hand. Such a method provides an intuitive way of feeling the force, while avoiding the instability issues, since the control loop in not closed from the slave to the master of the same hand. In the introductory paper [1], the technique was implemented using game controllers with simple tasks. In this paper, the technique was implemented on the da Vinci surgical system (Classic version) using the da Vinci Research Kit (dVRK) controllers that enable complete access to all control levels of the da Vinci robot manipulators via custom mechatronics and open-source software. The implementation involved a full re-write of a teleoperation controller based on kinematic correspondence with gravity compensation, as well as torque control functions for force rendering on the da Vinci master manipulators. A series of suture knot tying and haptic exploration experiments were conducted in which a small group of users, both surgeons (N=3) and novices (N=6) evaluated the system. The results show that the proposed technique has some promise when implemented in a realistic 14 degrees of freedom system, but further work is necessary to make the system fully usable.

Registration of histological slices to volumetric imaging of the prostate is an important task that can be used to optimize imaging for cancer detection. Such registration is challenging due to physical changes of the specimen during excision and fixation, and misalignment of the histological slices during preparation and digital scanning. In this work, we consider a multi-slice to volume registration method in which a stack of sparse, unaligned 2-D whole-mount histological slices is registered to a 3-D volumetric imaging of the prostate. We propose a particle filtering framework to contend with the high dimensionality of the search space and multimodal nature of the optimization. Such framework allows modeling of the uncertainty in the pose of the slices and in the imaged information, in order to derive optimal registration parameters in a Bayesian approach. Intensity-, region-, and point-based similarity metrics were incorporated into the registration algorithm to account for different imaging modalities. We demonstrate and evaluate our method on a diverse set of data that includes a synthetic volume, ex vivo and in vivo magnetic resonance imaging, and in vivo ultrasound.

Lack of haptic feedback in teleoperated robot-assisted surgery (RAS) is known to be detrimental in many surgical tasks. While performing a class of force sensitive tasks, surgeons commonly use both hands. Oftentimes, one hand is used to exert tension/compression forces, and the other to hold the suture knot or tissue. A novel control framework to accomplish haptic force feedback for two-handed tasks in teleoperated RAS is presented in this paper. The force applied on the surgical environment by the action hand is not transferred back to the same hand, but rather to the other hand. In two-handed tasks that involve an action and a reaction force, this provides an intuitive way of feeling the action. Because the loop in not closed from the slave back to the master of the same hand, it does not have a destabilizing effect. The technique can be easily implemented using a variable-structure controller that combines two PD controllers and a switch. It has been evaluated with an experimental setup consisting of four haptic devices with promising results.

The finite element method is commonly used to model tissue deformation in order to solve for unknown parameters in the inverse problem of viscoelasticity. Typically, a (regular-grid) structured mesh is used since the internal geometry of the domain to be identified is not known a priori. In this work, the generation of problem-specific meshes is studied and such meshes are shown to significantly improve inverse-problem elastic parameter reconstruction. Improved meshes are generated from axial strain images, which provide an approximation to the underlying structure, using an optimization-based mesh adaptation approach. Such strain-based adapted meshes fit the underlying geometry even at coarse mesh resolutions, therefore improving the effective resolution of the reconstruction at a given mesh size/complexity. Elasticity reconstructions are then performed iteratively using the reflective trust-region method for optimizing the fit between estimated and observed displacements. This approach is studied for Young's modulus reconstruction at various mesh resolutions through simulations, yielding 40%-72% decrease in root-mean-square reconstruction error and 4-52 times improvement in contrast-to-noise ratio in simulations of a numerical phantom with a circular inclusion. A noise study indicates that conventional structured meshes with no noise perform considerably worse than the proposed adapted meshes with noise levels up to 20% of the compression amplitude. A phantom study and preliminary in vivo results from a breast tumor case confirm the benefit of the proposed technique. Not only conventional axial strain images but also other elasticity approximations can be used to adapt meshes. This is demonstrated on images generated by combining axial strain and axial-shear strain, which enhances lateral image contrast in particular settings, consequently further improving mesh-adapted reconstructions.

In elasticity imaging, the shear modulus is obtained from measured tissue displacement data by solving an inverse problem based on the wave equation describing the tissue motion. In most inversion approaches, the wave equation is simplified using local homogeneity and incompressibility assumptions. This causes a loss of accuracy and therefore imaging artifacts in the resulting elasticity images. In this paper we present a new curl-based finite element method inversion technique that does not rely upon these simplifying assumptions. As done in previous research, we use the curl operator to eliminate the dilatational term in the wave equation, but we do not make the assumption of local homogeneity. We evaluate our approach using simulation data from a virtual tissue phantom assuming time harmonic motion and linear, isotropic, elastic behavior of the tissue. We show that our reconstruction results are superior to those obtained using previous curl-based methods with homogeneity assumption. We also show that with our approach, in the 2-D case, multi-frequency measurements provide better results than single-frequency measurements. Experimental results from magnetic resonance elastography of a CIRS elastography phantom confirm our simulation results and further demonstrate, in a quantitative and repeatable manner, that our method is accurate and robust.

In this study we evaluated a new method for registering three-dimensional ultrasound (3DUS) data to external coordinate systems. First, 3DUS was registered to the stereo endoscope of a da Vinci Surgical System by placing a registration tool against an air-tissue boundary so that the 3DUS could image ultrasound fiducials while the stereo endoscope could image camera markers on the same tool. The common points were used to solve the registration between the 3DUS and camera coordinate systems. The target registration error (TRE) when imaging through a PVC tissue phantom ranged from 3.85 1.76 mm to 1.82 1.03 mm using one to four registration tool positions. TRE when imaging through an ex-vivo liver tissue sample ranged from 2.36 1.01 mm to 1.51 0.70 mm using one to four registration tool positions. Second, using a similar method, 3DUS was registered to the kinematic coordinate system of a da Vinci Surgical System by using the da Vinci surgical manipulators to identify common points on an air-tissue boundary. TRE when imaging through a PVC tissue phantom was 0.95 0.38 mm. This registration method is simpler and potentially more accurate than methods using commercial motion tracking systems. This method may be useful in the future in augmented reality systems for laparoscopic and robotic-assisted surgery.

We aim to compute the movement of permanent stranded implant brachytherapy radioactive sources (seeds) in the prostate from the planned seed distribution to the intraoperative fluoroscopic distribution, and then to the postimplant computed tomography (CT) distribution. We present a novel approach to matching the seeds in these distributions to the plan by grouping the seeds into needle tracks. First, we identify the implantation axis using a sample consensus algorithm. Then, we use a network flow algorithm to group seeds into their needle tracks. Finally, we match the needles from the three stages using both their transverse plane location and the number of seeds per needle. We validated our approach on eight clinical prostate brachytherapy cases, having a total of 871 brachytherapy seeds distributed in 193 needles. For the intraoperative and postimplant data, 99.31% and 99.41% of the seeds were correctly assigned, respectively. For both the preplan to fluoroscopic and fluoroscopic to CT registrations, 100% of the needles were correctly matched. We show that there is an average intraoperative seed displacement of 4.94±2.42 mm and a further 2.97±1.81 mm of postimplant movement. This information reveals several directional trends and can be used for quality control, treatment planning, and intraoperative dosimetry that fuses ultrasound and fluoroscopy.

Prostate segmentation in B-mode images is a challenging task even when done manually by experts. In this paper we propose a 3D automatic prostate segmentation algorithm which makes use of information from both ultrasound B-mode and vibro-elastography data.We exploit the high contrast to noise ratio of vibro-elastography images of the prostate, in addition to the commonly used B-mode images, to implement a 2D Active Shape Model (ASM)-based segmentation algorithm on the midgland image. The prostate model is deformed by a combination of two measures: the gray level similarity and the continuity of the prostate edge in both image types. The automatically obtained mid-gland contour is then used to initialize a 3D segmentation algorithm which models the prostate as a tapered and warped ellipsoid. Vibro-elastography images are used in addition to ultrasound images to improve boundary detection.We report a Dice similarity coefficient of 0.87±0.07 and 0.87±0.08 comparing the 2D automatic contours with manual contours of two observers on 61 images. For 11 cases, a whole gland volume error of 10.2±2.2% and 13.5±4.1% and whole gland volume difference of -7.2±9.1% and -13.3±12.6% between 3D automatic and manual surfaces of two observers is obtained. This is the first validated work showing the fusion of B-mode and vibro-elastography data for automatic 3D segmentation of the prostate.

Vision-based tracking of tissue is a key component to enable augmented reality during a surgical operation. Conven- tional tracking techniques in computer vision rely on identifying strong edge features or distinctive textures in a well-lit environ- ment; however endoscopic tissue images do not have strong edge features, are poorly lit and exhibit a high degree of specular reflection. Therefore, prior work in achieving densely populated 3D features for describing tissue surface profiles require complex image processing techniques and have been limited in providing stable, long-term tracking or real-time processing. In this paper, we present an integrated framework for ac- curately tracking tissue in surgical stereo-cameras at real-time speeds. We use a combination of the STAR feature detector and Binary Robust Independent Elementary Features to acquire salient features that can be persistently tracked at high frame rates. The features are then used to acquire a densely-populated map of the deformations of tissue surface in 3D. We evaluate the method against popular feature algorithms in in-vivo animal study video sequences, and we also apply the proposed method to human partial nephrectomy video sequences. We extend the salient feature framework to support region tracking in order to maintain the spatial correspondence of a tracked region of tissue or a medical image registration to the surrounding tissue. In-vitro tissue studies show registration accuracies of 1.3-3.3 mm using a rigid-body transformation method.

This paper presents a haptic simulator for prostate brachytherapy. Both needle insertion and the manipulation of the transrectal ultrasound (TRUS) probe are controlled via haptic devices. These are used to render tissue interaction forces computed using a deformable tissue model based on the finite element method (FEM). Needle flexibility and lateral needle bevel forces are also simulated. The TRUS-tissue simulation allows a trainee to practice the 3D intra-operative placement of the TRUS probe for registration with the pre-operative volume study. The needle-tissue simulation allows a trainee to practice needle insertion and targeting. The TRUS probe and the needle can be maneuvered simultaneously. Approaches to computational acceleration for real-time haptic performance are presented. Trade-offs between accuracy and speed are discussed. A graphics-card implementation of the numerically intensive mesh-adaptation operation is also presented.

In this paper, a novel approach to the problem of elasticity reconstruction is introduced. In this approach, the solution of the wave equation is expanded as a sum of waves travelling in different directions sharing a common wave number. In particular, the solutions for the scalar and vector potentials which are related to the dilatational and shear components of the displacement respectively are expanded as sums of travelling waves. This solution is then used as a model and fitted to the measured displacements. The value of the shear wave number which yields the best fit is then used to find the elasticity at each spatial point. The main advantage of this method over direct inversion methods is that, instead of taking the derivatives of noisy measurement data, the derivatives are taken on the analytical model. This improves the results of the inversion. The dilatational and shear components of the displacement can also be computed as a byproduct of the method, without taking any derivatives. Experimental results show the effectiveness of this technique in magnetic resonance elastography. Comparisons are made with other state-of-the-art techniques.

In medical simulations involving tissue deformation, the finite element method (FEM) is a widely used technique, where the size, shape, and placement of the elements in a model are important factors that affect the interpolation and numerical errors of a solution. Conventional model generation schemes for FEM consist of a segmentation step delineating the anatomy followed by a meshing step generating elements conforming to this segmentation. In this paper, a single-step model generation technique is proposed based on optimization. Starting from an initial mesh covering the domain of interest, the mesh nodes are adjusted to minimize an objective function which penalizes intra-element intensity variations and poor element geometry for the entire mesh. Trade-offs between mesh geometry quality and intra-element variance are achieved by adjusting the relative weights of the geometric and intensity variation components of the cost function. This meshing approach enables a more accurate rendering of shapes with fewer elements and provides more accurate models for deformation simulation, especially when the image intensities represent a mechanical feature of the tissue such as the elastic modulus. The use of the proposed mesh optimization is demonstrated in 2-D and 3-D on synthetic phantoms, MR images of the brain, and CT images of the kidney. A comparison with previous meshing techniques that do not account for image intensity is also provided demonstrating the benefits of our approach.

In this paper, we propose an interpolation-based method for simulating rigid needles in B-mode ultrasound images in real time. We parameterize the needle B-mode image as a function of needle position and orientation. We collect needle images under various spatial configurations in a water-tank using a needle guidance robot. Then we use multidimensional tensor-product interpolation to simulate images of needles with arbitrary poses and positions using collected images. After further processing, the interpolated needle and seed images are superimposed on top of phantom or tissue image backgrounds. The similarity between the simulated and the real images is measured using a correlation metric. A comparison is also performed with in vivo images obtained during prostate brachytherapy. Our results, carried out for both the convex (transverse plane) and linear (sagittal/para-sagittal plane) arrays of a trans-rectal transducer indicate that our interpolation method produces good results while requiring modest computing resources. The needle simulation method we present can be extended to the simulation of ultrasound images of other wire-like objects. In particular, we have shown that the proposed approach can be used to simulate brachytherapy seeds.

This paper presents an algorithm for fast image synthesis inside deformed volumes. Given the node displacements of a mesh and a reference 3-D image dataset of a predeformed volume, the method first maps the image pixels that need to be synthesized from the deformed configuration to the nominal predeformed configuration, where the pixel intensities are obtained easily through interpolation in the regular-grid structure of the reference voxel volume. This mapping requires the identification of the mesh element enclosing each pixel for every image frame. To accelerate this point location operation, a fast method of projecting the deformed mesh on image pixels is introduced in this paper. The method presented was implemented for ultrasound B-mode image simulation of a synthetic tissue phantom. The phantom deformation as a result of ultrasound probe motion was modeled using the finite element method. Experimental images of the phantom under deformation were then compared with the corresponding synthesized images using sum of squared differences and mutual information metrics. Both this quantitative comparison and a qualitative assessment show that realistic images can be synthesized using the proposed technique. An ultrasound examination system was also implemented to demonstrate that real-time image synthesis with the proposed technique can be successfully integrated into a haptic simulation.

This paper presents a new needle path planning method for the insertion of rigid needles into deformable tissue. The needle insertion point, needle heading, and needle depth are optimized by minimizing the distance between a rigid needle and a number of targets in the tissue. The optimization method is based on iterative simulations performed using a tissue finite element model. At each iteration, the best 3-D line fitted to the displaced targets in the deformed tissue is used as a candidate for a new insertion line. First, this method is implemented in a prostate brachytherapy simulator under different boundary conditions to minimize the targeting error. It is shown that the optimization method converges in a few iterations and decreases the seed misplacement error to less than the needle diameter. Second, the efficacy of the optimization algorithm is verified by optimizing the insertion parameters for a brachytherapy needle before insertion into a prostate tissue phantom. The elastic properties of the phantom and the needle-tissue interaction parameters were identified in an independent experiment. The optimization algorithm is effective in decreasing the targeting error.

Low dose rate prostate brachytherapy involves the permanent implant of radioactive sources into the prostate region using needles. We present a four-degree-of-freedom robot for prostate brachytherapy. The robot can translate a needle guide in the X-Y plane allowing for precise needle insertion along the Z direction. It can also rotate the guide about the X and Y axes, providing thus fine control over the needle insertion point and angle. The robot is light and mountable on a standard brachytherapy stepper. It is non- back-drivable providing a stable needle insertion direction when the power is off. It allows for manual control of each of the motor axes for fine positioning and has a quick-release mechanism for gross translation of the needle guide. We present the robot design and the performance characteristics of the prototype we built. The robot has an interface that allows the guide to be stepped through a complete treatment plan. A radiation oncologist implanted a phantom according to a treatment plan. Only 32 minutes were required for the complete implant that had 26 needles with 136 seeds. This demonstrates that, in spite of its increased flexibility of use, the robotic guide does not add to the procedure time.

This paper presents a new method of optimizing the needle insertion point, heading and depth for needle insertion into deformable tissue. The goal is to minimize the distance between a number of specified targets and the needle. Assuming a rigid needle and a deformable tissue described by a finite element model, an iterative optimization method is proposed that uses the needle insertion simulation. At each iteration, the best fitted 3D line to the targets in the simulated deformed configuration is used as a candidate for the new insertion line in the next iteration. This method has been implemented in a prostate brachytherapy simulator to minimize seed misplacement errors. The targets are designed to lie on a straight line in the undeformed configuration inside the prostate. To increase the accuracy while simulating the prostate rotation, a non-linear model is used. The neo-Hookean material model is exploited to determine the effects of geometric and mechanical nonlinearities and compressibility effects. It is shown that the optimization algorithm converges in few iterations and decreases the targeting error effectively.

A method for vessel segmentation and tracking in ultrasound images using Kalman filters is presented. A modified Star-Kalman algorithm is used to determine vessel contours and ellipse parameters using an extended Kalman filter with an elliptical model. The parameters can be used to easily calculate the transverse vessel area which is of clinical use. A temporal Kalman filter is used for tracking the vessel center over several frames, using location measurements from a handheld sensorized ultrasound probe. The segmentation and tracking have been implemented in real-time and validated using simulated ultrasound data with known features and real data, for which expert segmentation was performed. Results indicate that mean errors between segmented contours and expert tracings are on the order of 1%-2% of the maximum feature dimension, and that the transverse cross-sectional vessel area as computed from estimated ellipse parameters a, b as determined by our algorithm is within 10% of that determined by experts. The location of the vessel center was tracked accurately for a range of speeds from 1.4 to 11.2 mm/s.

This paper presents an approach for haptic rendering of topological constraints to users operating serial virtual linkages. In the proposed approach, a haptic device controller is designed to penalize users’ departure from the configuration manifold of the virtual linkage. This manifold is locally approximated through the range space of the Jacobian of the virtual linkage computed at the user’s hand. Simulations and controlled experiments performed using a planar haptic interaction system demonstrate that the proposed approach successfully constrains the users’ motion as required by the topology of the virtual linkage that they manipulate.

A new simulation approach is proposed to improve the stability and the perceived rigidity of contacts during haptic interaction with multirigid body virtual environments. The approach computes impulsive forces upon contact and penalty and friction forces during contact. The impulsive forces are derived using a new multiple collision resolution method that never increases the kinetic energy of the system. When new contacts arise, the impulsive forces generate large hand accelerations without requiring increased contact stiffness and damping. Virtual objects and linkages are regarded as points in the configuration space, and no distinction is made between them in the proposed approach.

This paper addresses position and force control of robotic manipulators that are in contact with environments that exhibit mechanical impedances covering a large continuous range, from very soft to very stiff. For robot programming, automation, teleoperation and haptics, such environments enforce a trade-off between position and force control, which can be accommodated by impedance controllers. In this work, we extend the concept of duality and consider impedance matching in order to optimise a combined position and force trajectory error metric. The analysis of numerical optimisation results provides clear guidance on the choice of target impedance parameters, based on environment and manipulator dynamics.

Transparent teleoperation under rate mode has proven to be difficult in terms of stability, performance and implementation. This is mainly due to the need for the exchange of derivatives and integral of measured positions and forces. This paper proposes a new control architecture designed based on the environment impedance reflection concept. The performance of this controller is compared to that of a conventional controller, under different operational conditions using both analytical methods and numerical simulations.

A methodology for estimating the force distribution that occurs along a needle shaft during insertion is described. An experimental system for measuring planar tissue phantom deformation during needle insertions has been developed and is presented. A two-dimensional linear elastostatic material model, discretised using the finite element method, is used to derive contact force information that is not directly measurable. This approach provides a method for quantifying the needle forces and soft tissue deformations that occur during general needle trajectories in multiple dimensions. The needle force distribution is used for graphical and haptic real-time simulations of needle insertion. Since the force displacement relationship is required only along the needle, a condensation technique is shown to achieve very fast interactive simulations. Stiffness matrix changes corresponding to changes in boundary conditions and material coordinate frames are performed using low-rank matrix updates.

A methodology for estimating the force distribution that occurs along a needle shaft during insertion is described. To validate the approach, an experimental system for measuring planar tissue deformation during needle insertions has been developed and is presented. The planar motion of a soft tissue phantom is measured during needle penetration and used in conjunction with a two dimensional linear elastostatic material model, discretised using the finite element method, to derive contact force information that is not directly measurable. This approach provides a method for quantifying the needle forces and soft tissue deformations that occur during general needle trajectories in multiple dimensions. A simulation algorithm that is based upon these results is also described.

A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. The paper presents the visual servo controller used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image features that can be selected by the operator are recognized and tracked by a variety of techniques. Based on feature tracking, ultrasound image servoing in three axes has been incorporated in the interface and can be enabled to automatically compensate, through robot motions, unwanted motions in the plane of the ultrasound beam. The accuracy of the system is illustrated through a 3-D reconstruction of an ultrasound phantom. An Internet-based robot-assisted teleultrasound system has also been demonstrated.

This paper first investigates the issue of transparency in time-delayed teleoperation. It then studies the advantages of employing local force feedback for enhanced stability and performance. In addition, two classes of three-channel control architectures, that are perfectly transparent under ideal conditions are introduced. The stability robustness of the proposed architectures to delays is rigorously analyzed, leading to certain bounds on force feedforward control parameters. Experimental results are included in support of the theoretical work.

A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. This paper presents key features of the user interface used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image features that can be selected by the operator are recognized and tracked by a variety of techniques. Based on feature tracking, ultrasound image servoing in three axes has been incorporated in the interface and can be enabled to automatically compensate, through robot motions, unwanted motions in the plane of the ultrasound beam. The stability and accuracy of the system is illustrated through a 3D reconstruction of an ultrasound phantom.

An adaptive nonlinear controller is proposed for position tracking in hydraulic robots by adopting the backstepping methodology. Full system models including rigid body and hydraulic dynamics are incorporated in the design. The tracking errors are shown to be asymptotically stable in the presence of parametric uncertainties using Lyapunov analysis. The controller is applied to a Stewart type hydraulic manipulator. Simulation and experimental results are provided to demonstrate the effectiveness of the approach.

This paper addresses the control problem of hydraulic robot manipulators. The backstepping design methodology is adopted to develop a novel nonlinear position tracking controller. The tracking errors are shown to be exponentially stable under the proposed control law. The controller is further augmented with adaptation laws to compensate for parametric uncertainties in the system dynamics. Acceleration feedback is avoided by using two new adaptive and robust sliding-type observers. The adaptive controllers are proven to be asymptotically stable via Lyapunov analysis. Simulation and experimental results performed with a hydraulic Stewart platform demonstrate the effectiveness of the approach.

This paper presents the development of a novel, fully-automatic tracking and segmentation system to extract the boundary of the carotid artery from ultrasound images in real-time. The center of the carotid artery is tracked using the Star algorithm. The stability of the Star algorithm has been improved by using a temporal Kalman filter. A spatial Kalman filter is used to estimate the carotid artery boundary. Since the method does not employ any numerical optimization, convergence is very fast. The stability and accuracy of the method is demonstrated by tracking the carotid artery over a 30 second sequence of ultrasound images taken during a carotid artery examination. An application of the tracking method to ultrasound image servoing is also presented.

We present a system for rendering planar rigid-body motion by means of a redundant parallel mechanism. The device design, the control architecture and the passive virtual environment simulation are presented. The system is used to compare various virtual walls and friction models proposed for haptic applications. In addition, the reset-integrator dry friction model proposed by Haessig and Friedland (1991) is implemented in a haptic interface for the first time.

Teleoperation systems are subject to operator and environment dynamic uncertainties as well as communication-channel delays. For the first time in the context of teleoperation, the passivity-based Llewellyn's two-port network absolute stability criterion as well as the minima and the dynamic ranges (Z-widths) of the operator and environment transmitted impedances are employed to analyze stability and performance robustness of two and four channels bilateral control architectures. The results of these evaluations and the above analysis tools provide a framework for robust bilateral controller design.

A haptic interface controlled in impedance mode should present its user with a uniform force capability matching the human hand. This paper reviews a mechanism design methodology that maximizes a workspace-inclusive isotropy index. Force maximization in coreless or voice-coil motors is also considered in order to maximize haptic interface acceleration. Example designs of a planar 3-DOF haptic mouse, a 5-DOF haptic pen and a MagLev joystick are presented.

Describes the haptic rendering of a virtual environment by drawing upon concepts developed in the area of teleoperation. A four-channel teleoperation architecture is shown to be an effective means of coordinating the control of a 3-DOF haptic interface with the simulation of a virtual dynamic environment.

The control problem of a hydraulic servo-system is addressed. The performance achievable by classical linear controllers, e.g. PD, are usually limited due to the highly nonlinear behavior of the hydraulic dynamics. This paper adopts the backstepping design strategy to develop a Lyapunov-based nonlinear controller for a hydraulic servo-system. Load, hydraulic and valve dynamics are incorporated in the design process. An adaptation law is also proposed to deal with uncertainties in hydraulic parameters. The approach can be further extended to the control of hydraulically driven manipulators. Both simulation and experimental results are provided to show the effectiveness of the proposed method.

The authors previously (1999) presented a fully counter-balanced 6-DOF robot for ultrasound carotid artery diagnosis. This paper presents the design of its controller. The controller is a velocity controller capable of incorporating position control, force control, and image based control, using shared control. Safety is a significant issue in the design of the control system. The counterbalanced mechanical design enables the use of small motor torques which result in 10N maximum force at the probe, and allows the probe to remain in its position in case of power failure. The robot can be positioned by an input device via motion/force control and by image feature based control. It can also be positioned by directly pushing its linkage in case that either the patient or the sonographer shows the slightest sign of discomfort. Feasibility experiments are reported.

This paper studies the advantages of employing local force feedback for enhanced stability and performance in teleoperation systems. It also shows how a class of three-channel architecture bilateral controllers can provide perfect transparency under ideal conditions. Furthermore, the robustness of the proposed architecture to the communication channel time-delays is analytically investigated. Experimental results are also included to support the theoretical work.

An adaptive motion/force control based approach is proposed for the bilateral teleoperation systems under both position and rate control with arbitrary motion/force scaling. The master and the slave are treated separately, and are subject to independent adaptive motion/force control. A model of the human operator is incorporated into the dynamics of the master robot, while a model of the environment is incorporated into the dynamics of the slave robot. Position/velocity tracking errors between the master and slave robots are shown to be in L/sub 2//spl cap/L/sub /spl infin//. The overall teleoperation system is equivalent to a free-floating mass plus a linear damper specified by the control and scaling parameters only. In comparison to previous teleoperation approaches, this approach possesses three novel features: 1) it is L/sub 2/ and L/sub /spl infin// stability guaranteed with motion/force tracking capability, 2) it can handle parameter uncertainties by applying independent parameter adaptation, and 3) it takes into account the full nonlinear dynamics of the master/slave robots. The validity of the theoretical results are verified by experiments.

A four-channel data transmission structure has been proposed in the literature to achieve "transparency" for master-slave teleoperator systems under both position and rate control. In this paper, experimental results for a one degree-of-freedom system are given to demonstrate that the proposed scheme possesses good stability and transparency under both position and rate control in contact with both flexible and rigid environments. A unilateral constraint correction under rate control is also discussed.

Good robot performance often relies upon the selection of design parameters that lead to a well conditioned Jacobian or impedance "design" matrix. In this paper, a new design matrix normalization technique is presented to handle the problem of nonhomogeneous physical units and to provide a means of specifying a performance based design goal. The technique pre- and post-multiplies a design matrix by scaling matrices corresponding to a range of joint and task space variables. The task-space scale factors are used to set the relative required strength or speed along any axes of end point motion while the joint-space scale factors are treated as free design parameters to improve isotropy through nonhomogeneous actuation. The effect of scaling on actual designs is illustrated by a number of design examples using a global search method previously developed by the authors.

A novel yet simple approach for experimental determination of the link (mass and inertia-related) parameters and friction coefficients is developed for a typical excavator arm. The parameters are needed for indirect measurement of the external forces, compensation for the link weights in the operator's hand in a force-feedback teleoperation setup, impedance control of the arm, simulation of the manipulator dynamics, and model-based fault detection. Treating the machine arm as an open kinematic chain, its dynamic equations are presented symbolically. The static torque equations are derived from these equations and the gravitational parameters are defined accordingly. A new method for decoupled estimation of the gravitational parameters from static experiments is presented. Furthermore, the arm dynamics are expressed in a form which is linear in the inertia and friction-related parameters. The results obtained show that the identified model predicts the joint torques, in both static and dynamic conditions, with a very good accuracy.

In order to facilitate the testing and evaluation of control strategies and operator environments designed for heavy duty hydraulic machines, an excavator simulator has been developed and is described in this paper. The simulator comprises an impedance model of the excavator arm, a model for the bucket-ground interaction forces, a graphical environment and a haptic interface. This paper describes the simulator components and their integration.

Presents the design of a cardiac tissue stabilizing device with fixating surgical arm, developed for open-heart cardiac surgery to stabilize anastomosis sites during off-pump coronary artery bypass grafting surgeries. The stabilization device is a "C"-shaped hollow ring that holds the heart using vacuum pressure. The arm is a seven link, 18-DOF, redundant, passive, serial linkage. As it attaches directly to the patient on the chest retractor, the arm has a stable mechanical ground protecting the patient from injury should they move during surgery. Measured forces and torques from two pigs' hearts were scaled to human magnitudes to determine stabilizer lifting and arm braking requirements.

This paper addresses issues of transparency and implementation of dual hybrid teleoperation. A method for automatically adjusting the master and slave impedances to match stiff and soft environments and to interpolate in between them is presented and evaluated using simulations. The application of this technique to the force-feedback control of a mini-excavator is also presented and discussed with supporting experimental results.

Good robot performance often relies upon the selection of design parameters that lead to a well conditioned Jacobian or impedance "design" matrix. In this paper a new design matrix normalization technique is presented to cope with the problem of nonhomogeneous physical units. The technique pre and post-multiplies a design matrix by diagonal scaling matrices corresponding to the range of joint and task space variables. In the case of the Jacobian, normalization leads to a practical interpretation of a robot's "characteristic length" as the desired ratio between maximum linear and angular force or velocity. The scale factors can also be used to set relative required strength or speed along any axes of end-point motion and/or can be treated as free design parameters to improve isotropy through asymmetric actuation. The effect of scaling on actual designs is illustrated by a number of design examples using a global search method previously developed by the authors.

This paper describes the modeling, simulation, and control of an inverted, ceiling-mounted Stewart platform, designed to be a one-person motion simulator. The dynamic equations of the Stewart platform are derived using the virtual work principle. It is shown by simulations that the leg dynamics can be neglected. A model of the electrohydraulic actuator is derived and then verified using experimental data. A linkspace pressure-feedback controller is proposed for high performance with good stability robustness. With the above controller, the small-motion position bandwidth of the platform can reach 9 Hz along the vertical axis for a payload of about 140 kg.

To achieve transparency for teleoperation in unknown or time varying environments a class of indirect adaptive bilateral control schemes is developed based on the Slotine-Li (1989) "composite adaptive control" schemes and the "impedance bilateral control" architecture presented by Hannaford (1989). The proposed controllers need master and slave position, velocity and acceleration measurements and require no force sensing. Numerical simulations are worked out to demonstrate the transparency and robustness of the controllers to time delay.

A remotely controlled microgripper has been developed as an end-effector for a six-degree-of-freedom force-reflecting motion-scaling system. The microgripper is small and lightweight, and features relatively high force and stroke compared to other designs. Furthermore, force sensing enables the accurate measurement and control of tool-tissue forces, and the emulation of different mechanical devices. This paper presents the design of the microgripper and describes some control methods that have been implemented.

A coarse-fine approach to the design of high fidelity haptic interfaces is proposed based on prior work and new psychophysics studies. The approach involves a fine-motion six-degree-of-freedom parallel Lorentz actuator mounted on a series/parallel coarse-motion stage and coupled through a compliant transmission. The frequency response of a simplified model is used to illustrate the advantages of the approach. The workspaces of three coarse-motion platforms are compared. A novel twin-elbow manipulator with all but one of the drive motors in the base is proposed for its simplicity and large workspace size.

Gravitational and friction terms play an important role in achieving high performance control of heavy-duty hydraulic machines, such as excavators. In this paper, a new approach for decoupled estimation of the gravitational parameters is presented. Static experiments are carried out with an instrumented computer-controlled mini excavator to estimate the gravitational parameters. Load pins are used for indirect measurement of the joint torques from cylinder reaction forces. It is investigated via experiments that bucket payload can be estimated with a 5% accuracy. Furthermore, direct measurement of the actuators' friction shows that considerable amount of static friction exists inside the cylinders that cannot be neglected.

A high-performance magnetic levitation haptic interface has been developed to enable the user to interact dynamically with simulated environments by holding a levitated structure and directly feeling its computed force and motion responses. The haptic device consists of a levitated body with six degrees of freedom and motion ranges of /spl plusmn/5 mm and /spl plusmn/3.5 degrees in all directions. The current device can support weights of up to 20 N and can generate a torque of 1.7 Nm. Control bandwidths of up to 50 Hz and stiffnesses from 0.01 to 23 N/mm have been achieved by the device using a digital velocity estimator and 1 KHz control on each axis. The response of the levitated device has been made successfully to emulate virtual devices such as gimbals and bearings as well as different dynamic interactions such as hard solid contacts, dry and viscous friction, and textured surfaces.

A four-channel data transmission structure has been suggested in the literature to achieve "transparency" for master-slave teleoperator systems under position control. In this paper, the result is generalized to include teleoperator systems that are under rate control or more general master-slave kinematic correspondence laws, such as a mixed position/rate mode. A one degree-of-freedom example is given to outline the design and analysis of such a system for transparency and stability.

This paper presents a new pneumatic actuator for applications requiring moderate, but accurate, force control, such as teleoperation masters and robot wrists. The actuator uses voice-coil flapper valves and low-friction cylinders. Its force is controlled by servoing differential pressure. Aspects of valve design, system modeling, controller design, and experimental results are presented.

A new approach to the design of teleoperation systems is presented. It is proposed that the teleoperation slave be a coarse-fine manipulator with a fine-motion wrist identical to the teleoperation master. By using a combination of position and rate control, such a system would require only small operator hand motions but would provide low mechanical impedance, high motion resolution and force feedback over a substantial volume. A new teleoperation system, consisting of a conventional manipulator and two identical magnetically levitated wrists, has been developed using this approach and is described in this paper. Aspects of mechanical, system and computational design are discussed. It is shown that the best way to position the slave is by decoupling position and rate control, with the conventional robot controlled in rate mode and its wrist in position mode. Kinesthetic feedback is achieved through wrist-level coordinated force control. Transparency is improved through feedforward of sensed hand forces to the master and environment forces to the slave. To maintain stability, the amount of damping in the system is controlled by the sensed environment forces. Experimental results demonstrating excellent performance are presented.

This paper presents an observer-based approach to the determination of environment forces acting on a rigid body. Actuation forces, measured positions and orientations and identification of inertial parameters are used for this purpose. It is also shown that environment force estimates can be used in turn to correct rigid-body velocity estimates generated by observers, leading to accurate, low-noise estimates during contact tasks. The approach, shown to also apply to general serial mechanisms, is supported by experimental data obtained with a magnetically levitated wrist.>

This paper presents a new six-degree-of-freedom pneumatic fine-motion device that can be used as a robot wrist, a teleoperation master, or in other fine-motion applications requiring high position resolution and force control. The device uses six pairs of externally pressurized mechanical bellows as a pneumatic suspension without actuator friction. Forces and torques are controlled by servoing differential pressure across each pair of bellows using novel high-flow flapper valves. Aspects of design and control are presented. Experimental results show that the valves employed have no hysteresis and have flow exceeding that of any single stage pneumatic valve currently available on the market, while the pneumatic-suspension device shows promise as a force-reflecting teleoperation master. >

This paper describes the design of a one-person hydraulic motion simulator with six degrees of freedom. An inverted, ceiling-mounted Stewart platform design enables the use of narrower hydraulic cylinders than required with floor-mounted designs, and significantly reduces hydraulic flow requirements and hence the cost. The paper details the platform geometry, shows the effectiveness of Newton's method in solving the direct kinematics, and describes a tested hydraulic system design. Issues of control and safety are also addressed. It is expected that this motion simulator will provide 9.8 m/s/sup 2/, 400/spl deg//s/sup 2/ accelerations and 1 m/s, 30/spl deg//s speeds to a 250 kg payload with 1 m, 45/spl deg/ displacements from a nominal center. Preliminary experiments at low-pressure have been encouraging.>

This paper presents design aspects of a force-reflecting, motion-scaling teleoperation system for use in microsurgery experiments. A coarse-macro-micro approach is proposed. Two magnetically levitated wrists-a macro-master and a micro-slave-would share a common base positioned at the surgical site by a coarse-motion transport robot. The discussion includes features of the proposed system, a detailed slave wrist description, and aspects of coordination and control for the multistage teleoperation.>

The authors describe a six degree-of-freedom active experiment isolation system designed to cancel out residual accelerations during zero-gravity parabolic flights. The isolation system consists of a fine-motion magnetic levitator whose stator is transported by a conventional coarse-motion stage or by a robot. The levitator uses wide-gap voice-coil actuators and has the dual purpose of isolating the experiment platform from aircraft vibrations and actively cancelling residual accelerations through feedback control. The robot tracks the levitated platform to keep the levitator's coils centered within their matching magnetic gaps. Aspects of system design, an analysis of the proposed control strategy. and simulation results are presented. Feasibility experiments using a PUMA 500 and a magnetically levitated robot wrist are discussed.>

The authors present a force-reflecting teleoperation system. The system consists of a conventional manipulator equipped with a magnetically levitated teleoperation master identical to the wrist. Aspects of mechanical, system, computational, and controller design are discussed. An approach to the design of force-reflecting teleoperation controllers is also discussed. Admittance matrices and the parameterization of all stabilizing teleoperation system controllers are used to guarantee teleoperation system performance and stability. Some preliminary experimental results are presented.>

A high-performance six-degree-of-freedom magnetically levitated fine-motion wrist with programmable compliance is described. Design considerations, a discussion of the major elements of the device, and issues of modeling, kinematics, dynamics, and control are presented. A prototype wrist which has been built and controlled successfully is discussed. Experimental results, including high bandwidth position control, compliant control, and the emulation of several mechanisms through software gain setting which establish the use of magnetically levitated robot wrists as an option for manipulation tasks requiring high precision and fine compliant motions, are included.>

The problem of controlling the end-point position of a redundant coarse-fine manipulator is investigated. Such a system typically consists of a coarse robot for approximate positioning and, fastened to the end of it, a fast and accurate fine-motion robotic device. For small motions for which a linearized model of the coarse-fine dynamics is assumed, it is shown that, under mild assumptions, a good disturbance rejection over a wide frequency range of the fine manipulator is sufficient to insure stability and good performance of the coarse-fine redundant manipulator. The fine robot controller can be designed by H/sup infinity / norm minimization, while the coarse robot controller need not be changed.>

Reference is made to a previous paper by the authors and A.P. Allan (1988) in which the motivation for and the design of a magnetically levitated robot wrist have been described. The kinematic and dynamic transformations that are required for the control of this 'magic' wrist are presented. The wrist's dynamics are formulated in terms of the Euler quaternion, and the advantages of such a formulation, as well as control schemes based on it, are discussed. The prototype magic wrist was successfully controlled with a simple PD (proportional differential) feedback compensator, both in position and compliance modes. The result obtained, including synthesis of several mechanisms, establish the feasibility of the approach proposed in the original paper.>