Surgical robots equipped with cable-driven mechanisms have flexible, light, and compact arms and tools. However, cable slack, stretch, and gear backlash introduce unavoidable errors from motor positions to joint positions and the end-effector pose. This paper presents a learning-based joint position calibration method for the RAVEN-II surgical robot, employing deep neural networks and gated recurrent units. Compared to fixed offset compensation, the learning-based calibrations reduce the joint position errors by over 62.4% (unloaded) and 54.8% (loaded). Furthermore, removal and inaccurate ablation studies on input features identify that raw joint positions and motor torques are the most important model inputs for calibration accuracy. These studies also reveal that the models are capable of inferring joint positions from the end-effector pose and prioritize the direction of motor torques over their amplitude. When guided appropriately, the models can also compensate for encoder value inconsistencies occurring with robot re-homings. By excluding the unnecessary input features, lightweight models are developed and achieve better performance and efficiency simultaneously, reducing the training time on the CPU to 2.5 minutes. All data and code are open-source at https://github.com/uw-biorobotics/RAVEN-2-Feature-Ablation.

Vision measurement is important for intelligent systems to obtain the precise structural and spatial information of objects. Beyond the object-specific vision measurement developed for fixed object type, it is appealing to explore the object-agnostic vision measurement, which can be efficiently reconfigured and adapted to various novel objects. This article proposes a framework to mimic the human's versatile visual measurement behavior: extract a set of contour primitives of interest (CPIs) from an image, then utilize the CPIs to calculate the key geometric information. First, a deep convolutional neural network (CNN) CPieNet+ is proposed under the one-shot learning scheme, aiming to extract the pixel-level object CPI from a raw query image, given an annotated support image. The fine-grained CPI prototypes are formed by sampling multiple points on the feature map of the support image. To leverage the explicit geometric knowledge in the CNN inference, the annotation map is encoded as a shape descriptor to guide the feature channel attention, and the geometric attribute awareness is realized by supervising the model to predict the direction and size of CPI. Second, the measurement behavior tree (BT) is designed to model the hierarchical geometric calculation procedure, which is flexibly configurable for different measurement requirements and is interpretable for nonexpert users. After the execution of the measurement BT, the pixel-level CPIs are converted to the required key geometric data. The effectiveness of the proposed methods is validated by a series of experiments.

Existing studies demonstrated that surgical motion patterns are strongly correlated with surgical outcomes. Real surgeries are complicated and it is expensive to harvest surgical data. Consequently, existing researches on surgical motion patterns focus on specific concise surgical tasks or simple surgical procedures. The paper presents a surgical motion pattern modeling technique that uses small data but can be applied to virtually any Endoscopic Sinus and Skull Base Surgeries (ESSBSs). The proposed method decreases the dimensionalities of the feature space through projecting surgical instrument motions into the endoscope coordinate, based on human expert domain knowledge. Furthermore, the method uses kinematic features and learns the motion pattern with Gaussian Process learning techniques. Comparing with existing surgical motion pattern modeling methods, the proposed method: 1, learns the motion model from small data; 2, can be generally applied to ESSBSs because it neither assumes nor depends on specific surgical tasks; 3, provides informative results in a real-time manner for optimizing surgical motions for improving surgical outcomes. The proposed method was verified by predicting surgical skill levels on cadaver surgeries. The results show the real-time prediction precision is higher than 81% and the offline accumulated precision reach 100%.

The International Space Station (ISS) serves as a research lab for a wide variety of experiments including some that study the biological effects of microgravity and spaceflight using the Rodent Habitat and Microgravity Science Glovebox (MSG). Astronauts train for onboard dissections of rodents following basic training. An alternative approach for conducting these experiments is teleoperation of a robot located on the ISS from earth by a scientist who is proficient in rodent dissection. This pilot study addresses (1) the effects of extreme time delay on skill degradation during Fundamentals of Laparoscopic Surgery (FLS) tasks and rodent dissections using RAVEN II; (2) derivation and testing of rudimentary interaction force estimation; (3) elicitation of design requirements for an onboard dissection robot, RAVEN-S; and (4) simulation of the RAVEN-S prototype design with dissection data. The results indicate that the tasks' completion times increased by a factor of up to 9 for a 3 s time delay while performing manipulation and cutting tasks (FLS model) and by a factor of up to 3 for a 0.75 s time delay during mouse dissection tasks (animal model). Average robot forces/torques of 14N/0.1Nm (peak 90N/0.75Nm) were measured along with average linear/angular velocities of 0.02m/s/4rad/s (peak 0.1m/s/40rad/s) during dissection. A triangular configuration of three arms with respect to the operation site showed the best configuration given the MSG geometry and the dissection tasks. In conclusion, the results confirm the feasibility of utilizing a surgically-inspired RAVEN-S robot for teleoperated rodent dissection for successful completion of the predefined tasks in the presence of communications time delay between the ISS and ground control.

Surgical robots have been introduced to operating rooms over the past few decades due to their high sensitivity, small size, and remote controllability. The cable-driven nature of many surgical robots allows the systems to be dexterous and lightweight, with diameters as low as 5mm. However, due to the slack and stretch of the cables and the backlash of the gears, inevitable uncertainties are brought into the kinematics calcu-lation [1]. Since the reported end effector position of surgical robots like RAVEN-II [2] is directly calculated using the motor encoder measurements and forward kinematics, it may contain relatively large error up to 10mm, whereas semi-autonomous functions being introduced into abdominal surgeries require position inaccuracy of at most 1mm. To resolve the problem, a cost-effective, real-time and data-driven pipeline for robot end effector position precision estimation is proposed and tested on RAVEN-II. Analysis shows an improved end effector position error of around 1mm RMS traversing through the entire robot workspace without high-resolution motion tracker. The open source code, data sets, videos, and user guide can be found at //github.com/HaonanPeng/RAVEN Neural Network Estimator.

Inverse kinematics solves the problem of how to control robot arm joints to achieve desired end effector positions, which is critical to any robot arm design and implementations of control algorithms. It is a common misunderstanding that closed-form inverse kinematics analysis is solved. Popular software and algorithms, such as gradient descent or any multi-variant equations solving algorithm, claims solving inverse kinematics but only on the numerical level. While the numerical inverse kinematics solutions are relatively straightforward to obtain, these methods often fail, even when the inverse kinematics solutions exist. Therefore, closed-form inverse kinematics analysis is superior, but there is no generalized automated algorithm. Up till now, the high-level logical reasoning involved in solving closed-form inverse kinematics made it hard to automate, so it's handled by human experts. We developed IKBT, a knowledge-based intelligent system that can mimic human experts' behaviors in solving closed-from inverse kinematics using Behavior Tree. Knowledge and rules used by engineers when solving closed-from inverse kinematics are encoded as actions in Behavior Tree. The order of applying these rules is governed by higher level composite nodes, which resembles the logical reasoning process of engineers. It is also the first time that the dependency of joint variables, an important issue in inverse kinematics analysis, is automatically tracked in graph form. Besides generating closed-form solutions, IKBT also explains its solving strategies in human (engineers) interpretable form. This is a proof-of-concept of using Behavior Trees to solve high-cognitive problems.

Intelligent vision is appealing in computer-assisted and robotic surgeries. Vision-based analysis with deep learning usually requires large labeled datasets, but manual data labeling is expensive and time-consuming in medical problems. We investigate a novel cross-domain strategy to reduce the need for manual data labeling by proposing an image-to-image translation model live-cadaver GAN (LC-GAN) based on generative adversarial networks (GANs). We consider a situation when a labeled cadaveric surgery dataset is available while the task is instrument segmentation on an unlabeled live surgery dataset. We train LC-GAN to learn the mappings between the cadaveric and live images. For live image segmentation, we first translate the live images to fake-cadaveric images with LC-GAN and then perform segmentation on the fake-cadaveric images with models trained on the real cadaveric dataset. The proposed method fully makes use of the labeled cadaveric dataset for live image segmentation without the need to label the live dataset. LC-GAN has two generators with different architectures that leverage the deep feature representation learned from the cadaveric image based segmentation task. Moreover, we propose the structural similarity loss and segmentation consistency loss to improve the semantic consistency during translation. Our model achieves better image-to-image translation and leads to improved segmentation performance in the proposed cross-domain segmentation task.

The real-time and robust surgical instrument segmentation is an important issue for endoscopic vision. We propose an instrument segmentation method fusing the convolutional neural networks (CNN) prediction and the kinematic pose information. First, the CNN model ToolNet-C is designed, which cascades a convolutional feature extractor trained over numerous unlabeled images and a pixel-wise segmentor trained on few labeled images. Second, the silhouette projection of the instrument body onto the endoscopic image is implemented based on the measured kinematic pose. Third, the particle filter with the shape matching likelihood and the weight suppression is proposed for data fusion, whose estimate refines the kinematic pose. The refined pose determines an accurate silhouette mask, which is the final segmentation output. The experiments are conducted with a surgical navigation system, several animal-tissue backgrounds, and a debrider instrument.

Deformable objects and surfaces are ubiquitous in the daily lives of humans - from the garments in fashion to soft tissues within the body. Because of this routine interaction with soft materials, humans are adept and trained in manipulation of deformable objects while avoiding irreversible damage. The dexterity and care involved is largely facilitated through a combination of the human haptic sense of touch and visual observations of object deformation [1]. While this scenario presents itself as a trivially intuitive task, it becomes significantly more difficult and complex with the deprivation of both 3D depth perception and haptic senses. This deprived state is not dissimilar to the scenarios encountered in many robot-assisted minimally invasive surgeries. As a result, unintentional tissue damage can occur due to lack of force feedback and fine 3D visibility [2]. One approach to remediate these issues combines real-time dynamic 3D reconstruction and vision-based force estimation for haptic feedback. Toward that end, this work continues research in a series of studies focusing on multicamera 3D reconstruction of dynamic surgical cavities. Previous work introduced a novel approach of camera grouping and pair sequencing [3]. This paper builds upon that work by introducing a method for non-rigid, sparse point cloud registration and subsequent point classification. In particular, to enable deformation and force analyses, surfaces are locally classified into three categories: static, shifting and deforming. The topics addressed in this paper present open challenges and ongoing research directions for researchers to this day [4], and provide a step towards real-time 3D reconstruction and force feedback in robot-assisted surgery.

Inverse kinematics solves the problem of how to control robot arm joints to achieve desired end effector positions, which is critical to any robot arm design and implementations of control algorithms. It is a common misunderstanding that closed-form inverse kinematics analysis is solved. Popular software and algorithms, such as gradient descent or any multi-variant equations solving algorithm, claims solving inverse kinematics but only on the numerical level. While the numerical inverse kinematics solutions are relatively straightforward to obtain, these methods often fail, due to dependency on specific numerical values, even when the inverse kinematics solutions exist. Therefore, closed-form inverse kinematics analysis is superior, but there is no generalized automated algorithm. Up till now, the high-level logical reasoning involved in solving closed-form inverse kinematics made it hard to automate, so it's handled by human experts. We developed IKBT, a knowledge-based intelligent system that can mimic human experts' behaviors in solving closed-from inverse kinematics using Behavior Tree. Knowledge and rules used by engineers when solving closed-from inverse kinematics are encoded as actions in Behavior Tree. The order of applying these rules is governed by higher level composite nodes, which resembles the logical reasoning process of engineers. It is also the first time that the dependency of joint variables, an important issue in inverse kinematics analysis, is automatically tracked in graph form. Besides generating closed-form solutions, IKBT also explains its solving strategies in human (engineers) interpretable form. This is a proof-of-concept of using Behavior Trees to solve high-cognitive problems.

Recently, Recurrent Neural Network (RNN) control schemes for redundant manipulators have been extensively studied. These control schemes demonstrate superior computational efficiency, control precision, and control robustness. However, they lack planning completeness. This paper explains why RNN control schemes suffer from the problem. Based on the analysis, this work presents a new random RNN control scheme, which 1) introduces randomness into RNN to address the planning completeness problem, 2) improves control precision with a new optimization target, 3) improves planning efficiency through learning from exploration. Theoretical analyses are used to prove the global stability, the planning completeness, and the computational complexity of the proposed method. Software simulation is provided to demonstrate the improved robustness against noise, the planning completeness and the improved planning efficiency of the proposed method over benchmark RNN control schemes. Real-world experiments are presented to demonstrate the application of the proposed method.

Recurrent Neural Networks (RNNs) demonstrated advantages on control precision, system robustness and computational efficiency, and have been widely applied to redundant manipulator control optimization. Existing RNN control schemes locally optimize trajectories and are efficient and reliable on obstacle avoidance. However, for motion planning, they suffer from local minimum and do not have planning completeness. This work explained the cause of the planning incompleteness and addressed the problem with a novel RNN control scheme. The paper presented the proposed method in detail and analyzed the global stability and the planning completeness in theory. The proposed method was compared with other three control schemes on the precision, the robustness and the planning completeness in software simulation and the results shows the proposed method has improved precision and robustness, and planning completeness.

Compressing soft-obstacles secondary to a controlled motion task is common for human beings. While these tasks are nearly trivial for teleoperated robots, they remain a challenging problem in robotic autonomy. Addressing the problem is significant. For example, in Minimally Invasive Surgeries (MISs), safely compressing soft tissues ensures the surgical safety and decreases tissue removal, thus dramatically decreases surgical trauma and operating room time, and leads to improved surgical outcomes. In this work, we define the problem of soft-obstacle avoidance and project the safety motion constraints into the task space and the velocity space. We illustrate the significance of addressing this problem in the robotic surgery scenario. We present a Recurrent Neural Networks (RNNs) based solution, which formulates the problem as an inequality constrained optimization problem and solves it in its dual space. The application of the proposed method was demonstrated in the Raven II surgical robot. Experimental results demonstrated that the proposed method is effective in addressing the soft-obstacle avoidance problem.

Hand gestures are a natural component of human-human communication. Simple hand gestures are intuitive and can exhibit great lexical variety. It stands to reason that such a user input mechanism can have many benefits, including seamless interaction, intuitive control and robustness to physical constraints and ambient electrical, light and sound interference. However, while semantic and logical information encoded via hand gestures is readily decoded by humans, leveraging this communication channel in human-machine interfaces remains a challenge. Recent data-driven deep learning approaches are promising towards uncovering abstract and complex relationships that manual and direct rule-based classification schemes fail to discover. Such an approach is amenable towards hand gesture recognition, but requires myriad data which can be collected physically via user experiments. This process, however, is onerous and tedious. A streamlined approach with less overhead is sought. To that end, this work presents a novel method of synthetic hand gesture dataset generation that leverages modern gaming engines. Furthermore, preliminary results indicate that the dataset, despite being synthetic and requiring no physical data collection, is both accurate and rich enough to train a real-world hand gesture classifier that operates in real-time.

3D reconstruction and surgical tool segmentation are necessary for several advanced tasks in robot-assisted laparoscopic surgery. These tasks include vision-based force estimation, surgical guidance, and medical image registration where pre-operative data (CT or MRI scan image slices) are overlaid on patient anatomy in real-time during surgery [1] to name a few. In this work, two main strategies were considered: (1) initialize with surgical tool segmentation from 2D images, then proceed to local 3D reconstruction near the tool-tissue interaction region by projecting the segmented result into 3D space, and (2) initialize with 3D reconstruction of the entire surgical task space, followed by surgical tool segmentation from within the 3D reconstructed model. Both methods were implemented on the Raven II surgical robot system, and accuracy and time complexity for both methods were comparatively analyzed while considering various task parameters. Finally, based on the results of this work, guidelines for selecting reconstruction and segmentation strategies and procedure for particular situations are outlined in Section V.

Minimally Invasive Surgery (MIS) can reduce iatrogenic injury and decrease the possibility of surgical complications. This paper presents a novel flexible and bendable endoscopic device, “Roboscope”, which delivers two instruments, two miniature scanning fiber endoscopes, and a suction/irrigation port to the operation site through a single portal. Compared with existing bendable and steerable robotic surgical systems, Roboscope provides two bending degrees of freedom for its outer sheath and two insertion degrees of freedom, while simultaneously delivering two instruments and two endoscopes to the surgical site. Each bending axis and insertion freedom of Roboscope is independently controllable via an external actuation pack. Surgical tools can be changed without retracting the robot arm. This paper presents the design of the Roboscope mechanical system, electrical system, and control and software systems, design requirements and prototyping validation as well as analysis of Roboscope workspece.

Beside the output-input hysteresis, the longitudinally loaded cables of medical robotics such as RAVEN II exhibit asymmetric saturated strain-load hysteresis loops. This study investigates modeling the hysteresis nonlinearities of these cables using a stop-operator based Prandtl-Ishlinskii (SPI) model that is integrated with a memoryless function. The stop-operator based model is employed to account for the hysteresis nonlinearities, while the memoryless function is introduced to characterize saturation and asymmetric effects. A numerical example is presented to compare the properties of the proposed model with the classic SPI model. The response of the suggested model was evaluated on the hysteresis properties of two different cables subjected to triangular harmonic input of 0 to 0.001 with 6.25 × 10-5strain/s. The characterization error of the thick cable was found as 1.55 %, while the error was calculated as 1.25 % for the thin cable. The relative significance of the proposed model was further examined by comparing the measured data with the classic SPI model. The results showed that the classic model yields substantial characterization errors when the asymmetry and saturation effects of the strain-load hysteresis loops are ignored.

Surgical robot research is driven by the desire of improving surgical outcomes. This paper proposed a Recurrent Neural Network based controller to address two problems: 1) improving control precision, 2) increasing adaptiveness for robot motion (explained in Section I). RNN was adopted in this work mainly because 1) the problem formulation naturally matches RNN structure, 2) RNN has advantages as an biologically inspired method. The proposed method was explained in detail and analysis shows that the proposed method is able to dynamically regulate outputs to increase the adaptiveness and the control precision. This paper uses Raven II surgical robot as an example to show the application of the proposed method, and the numeral simulation results from the proposed method and three other controllers show that the proposed method has improved precision, improved high robustness against noise and increased movement smoothness, and it keeps the manipulator links as far away as possible from physical boundaries, which potentially increases surgical safety and leads to improved surgical outcomes.

Cable driven manipulators are popular in surgical robots due to compact design, low inertia, and remote actuation. In these manipulators, encoders are usually mounted on the motor, and joint angles are estimated based on transmission kinematics. However, due to non-linear properties of cables such as cable stretch, lower stiffness, and uncertainties in kinematic model parameters, the precision of joint angle estimation is limited with transmission kinematics approach. To improve the positioning of these manipulators, we use a pair of low cost stereo camera as the observation for joint angles and we input these noisy measurements into an Unscented Kalman Filter (UKF) for state estimation. We use the dual UKF to estimate cable parameters and states offline. We evaluated the effectiveness of the proposed method on a Raven-II experimental surgical research platform. Additional encoders at the joint output were employed as a reference system. From the experiments, the UKF improved the accuracy of joint angle estimation by 33- 72%. Also, we tested the reliability of state estimation under camera occlusion. We found that when the system dynamics is tuned with offline UKF parameter estimation, the camera occlusion has no effect on the online state estimation.

Haptic feedback plays a key role in surgeries, but it is still a missing component in robotic Minimally Invasive Surgeries. This paper proposes a dynamic model-based sensorless grip force estimation method to address the haptic perception problem for commonly used elongated cable-driven surgical instruments. Cable and cable-pulley properties are studied for dynamic modeling; grip forces, along with driven motor and gripper jaw positions and velocities are jointly estimated with Unscented Kalman Filter and only motor encoder readings and motor output torques are assumed to be known. A bounding filter is used to compensate for model inaccuracy and to improve method robustness. The proposed method was validated on a 10mm gripper which is driven by a Raven-II surgical robot. The gripper was equipped with 1-dimensional force sensors which served as ground truth data. The experimental results showed that the proposed method provides sufficiently good grip force estimation, while only motor encoder and the motor torques are used as observations.

In this paper, we propose model of longitudinally loaded cable based on the Bouc-Wen hysteresis model and within the framework of the Duhem operator. By optimizing the 9 hysteresis model parameters with a genetic algorithm, the proposed model is shown to be capable of representing quasi-static response of two different diameter cables, 0.61 mm (thin) and 1.19 mm (thick), used for the RAVEN II surgical robotic surgery platform. The construction of the cable is 7 strands with 19 individual wires per strand. Furthermore, it is shown that the dynamic response of the cables are captured by adding a linear damping term. The hysteresis model and linear damper with the optimized parameters accurately models a longitudinal vibration test result in terms of frequency, steady state stretch, and logarithmic decrement. Energy dissipation due solely to the hysteresis term is approximately calculated to be 57 and 71% of the total energy loss for the thin and thick cables respectively. The proposed model may be used for cables with different contraction and diameter and can be applied for control of cable driven robots in which cables are stretched longitudinally without large excitation of other modes.

Medical robots have been widely used to assist surgeons to carry out dexterous surgical tasks via various ways. Most of the tasks require surgeon's operation directly or indirectly. Certain level of autonomy in robotic surgery could not only free the surgeon from some tedious repetitive tasks, but also utilize the advantages of robot: high dexterity and accuracy. This paper presents a semi-autonomous neurosurgical procedure of brain tumor ablation using RAVEN Surgical Robot and stereo visual feedback. By integrating with the behavior tree framework, the whole surgical task is modeled flexibly and intelligently as nodes and leaves of a behavior tree. This paper provides three contributions mainly: (1) describing the brain tumor ablation as an ideal candidate for autonomous robotic surgery, (2) modeling and implementing the semi-autonomous surgical task using behavior tree framework, and (3) designing an experimental simulated ablation task for feasibility study and robot performance analysis.

Cable driven power transmission is popular in many manipulator applications including medical arms. In spite of advantages obtained by removing motors from the mechanism, cable transmission introduces higher non-linearity and more uncertainties such as cable stretch and cable coupling. In order to improve the control precision and robustness of the Raven-II surgical robot, particularly for automation applications, the Unscented Kalman Filter (UKF) was adopted for state estimation. The UKF estimated state variables of the Raven-II dynamic model from sensor data. The dual UKF was used offline to estimate cable coupling parameters. The experimental results showed that the proposed method improved joint position estimation precision and the estimation consistency, especially on the more elastic links. The improvements for links 2 and 3 of the Raven were 36.76%, and 62.99%, respectively. For link 1 the improvement was 1.43% because the transmission is very stiff.

This paper considers the semi-automated robotic surgical procedure for removing the brain tumor margins, where the manual operation is a tedious and time-consuming task for surgeons. We present robust path planning methods for robotic ablation of tumor residues in various shapes, which are represented in point-clouds instead of analytical geometry. Along with the path plans, corresponding metrics are also delivered to the surgeon for selecting the optimal candidate in the automated robotic ablation. The selected path plan is then executed and tested on RAVEN™ II surgical robot platform as part of the semi-automated robotic brain tumor ablation surgery in a simulated tissue phantom.

In this paper we present experimentally obtained cable-pulley Coulomb and viscous friction for cable-driven surgical robotic systems including the RAVEN II surgical robotic research platform. In the study of controlling cable-driven systems a simple mathematical model which does not capture physical behavior well is often employed. Even though control of such systems is achievable without an accurate model, fully understanding the behavior of the system will potentially realize more robust control. A surgical robot is one of the systems that often relies on cables as an actuation method as well as pulleys to guide them. Systems with such structure encounter frictional force related to conditions of cable and pulley such as cable velocity, tension, type and number of pulley, and angle of cable wrapping around pulley. Using a couple of test platforms that incorporate cable, pulleys, and other experimental conditions corresponding to the RAVEN II system, it is shown that cable-pulley friction is function of tension, wrap angle, and number of pulleys and not of magnitude of cable velocity.

This paper presents a novel surgical instrument integrated with a four-degree-of-freedom (DOF) force sensor. By adopting the capacitive transduction principle, the sensor enables the direct sensing of normal and shear forces at surgical instrument tips. Thus, three-DOF pulling forces and a single-DOF grasping force can be measured for haptic feedback control of robotic minimally invasive surgery systems. The sensor consists of four capacitive transducers, and all the transducers including analog signal processing units are embedded in small surgical instrument tips. The four-DOF force sensing is enabled thanks to the four capacitive transducers by using the force transformation method. In this study, the instrument is designed and manufactured to be adaptable to the open-source surgical robot platform, called Raven-II. In addition, the sensing system is experimentally validated through its application to the Raven-II by using a reference force sensor.

Using an accelerometer on the base of a robot, it is possible to calculate the torque required from each actuator in order to maintain a known pose regardless of base orientation with respect to the direction or magnitude of gravity. A simple and novel method has been developed and implemented for overcoming gravity induced torques on the RAVENTMsurgical research robot. This innovation will allow for accurate control of serial robot manipulators with re-orientable bases or for those operating in non-stationary environments such as boats, space stations, or moving vehicles.

Telerobotic task performance cannot compare to direct object manipulation with the hands. However, the computer in-the-loop offers the potential to give assistance to a human operator. The present work studies a class of computer assistance functions known as haptic virtual fixtures (VF). The objective is to use practical human trials to discover the ways VFs impact task execution.

Vibration motors are often used to generate tactile feedback to enhance human-machine interactions and provide information about the environment. We are interested in using these motors to enhance user feedback when wearing below-knee prostheses by providing informational cues via vibrations on the thigh. Our initial designs to hold the motors against the thigh resulted in a weak perception of vibration. We took an engineering approach to improve sensation by modeling the system and designing a new device that maximized skin displacement. Our results show the new suspended design increased skin displacement for both types of vibrational motors.

Vibration motors are often used to generate tactile effects by exciting a mass at a given frequency and amplitude. The characteristic impedance of this vibrotactile device is not always in harmony with the impedance of the human skin. This impedance mismatch can result in poor energy transfer, necessitating larger motors and greater power consumption than otherwise required. Herein, we investigate the feasibility of improving the energy transfer by placing a medium between the skin and the motor, which we dub an impedance adapter. We simulate the effects of this impedance adapter using a mathematical model, and evaluate its effect on skin displacement and a parameter we call skin stimulus. Skin stimulus is introduced as a measure of the perceptive effects of a haptic system, and is used to compare results between systems with an impedance adapter and those without. Our findings suggest a factor of four improvement in skin displacement and a two-fold increase in skin stimulus are possible by introducing an optimized impedance adapter.

Vibrotactile devices suffer from poor energy efficiency, arising from a mismatch between the device and the impedance of the human skin. This results in over-sized actuators and excessive power consumption, and prevents development of more sophisticated, miniaturized and low-power mobile tactile devices. In this paper, we present the experimental evaluation of a vibrotactile system designed to match the impedance of the skin to the impedance of the actuator. This system is able to quadruple the motion of the skin without increasing power consumption, and produce sensations equivalent to a standard system while consuming 1/2 of the power. By greatly reducing the size and power constraints of vibrotactile actuators, this technology offers a means to realize more sophisticated, smaller haptic devices for the user interface community.

This paper investigates automation of soft tissue compression for robot-assisted surgery. This is a fundamental task in surgery and includes interaction with a variety of tissues with unknown properties. In addition, due to sterilization and size constraints the use of contact force and position sensors are often avoided in surgical applications. We propose an Adaptive Model Predictive Control approach for execution of given tool trajectories in contact with unknown tissues in the absence of contact measurements. The Unscented Kalman Filter is employed in advance of system operation to identify the dynamics of a cable driven manipulator. These dynamics are then used to estimate contact force and position in free motion and in contact with tissue. An optimal control problem for automating tissue compression is formulated and is solved in real-time using Differential Dynamic Programming with Automatic Differentiation. The proposed methods are evaluated in experiments on an artificial tissue sample with unknown properties.

Cable driven power transmissions are used in applications such as haptic devices, surgical robots etc. The use of flexible cable based power transmission often causes relative motion between the motor actuator and mechanism joint during operation due to the elasticity of the cable. State-space control methods can be used to improve performance, but may require state estimates. For nonlinear systems, the Unscented Kalman Filter (UKF) provides a computationally efficient way to obtain state estimates. The UKF is applied here to a simulation of a minimially invasive surgical robot, to study the state estimation for a cable driven system with nonlinear dynamics. State estimates from the UKF are compared with the known states available from the simulation. These state estimates are also utilized by two different controllers interacting with the simulation to test the UKF performance under closed loop control. We tested the UKF performance with error perturbations in the system model's cable stiffness parameter.

In the task-saturated environment of extravehicular activity (EVA), an astronaut's ability to leverage suit-integrated information systems is limited by a lack of options for data entry. In particular, bulky gloves inhibit the ability to interact with standard computing interfaces such as a mouse or keyboard. This paper presents the results of a preliminary investigation into a system that permits the space suit gloves themselves to be used as data entry devices. Hand motion tracking is combined with simple finger gesture recognition to enable use of a virtual keyboard, while tactile feedback provides touch-based context to the graphical user interface (GUI) and positive confirmation of keystroke events. In human subject trials, conducted with twenty participants using a prototype system, participants entered text significantly faster with tactile feedback than without (p = 0.02). The results support incorporation of vibrotactile information in a future system that will enable full touch typing and general mouse interactions using instrumented EVA gloves.

Electric DC motors that convert electric current to torque are the most common type of actuators used in haptic interfaces. However, high currents necessary to deliver large haptic forces can generate heat in the electromagnetic coils, and in the extreme can cause malfunctions due to overheating. It is therefore necessary to add thermal protection mechanisms to limit the output current. While this is a common feature of haptic devices, it is not clear what makes a superior current limiting method or what effect such limits have on haptic perception. The current work evaluates three methods for current limiting: limit output to zero, limit output to a safe steady-state current, limit output current in linear proportion to temperature. Using both quantitative and qualitative metrics, these are compared to a control case with no current limit. Human subjects use one finger of a multi finger haptic device to perform a psychophysical thresholding experiment designed to measure human perception of small haptic effects, while forcing the device to heat up. All methods are shown to effectively regulate temperature and show no statistically significant difference in psychophysical threshold value. Users show a slight preference for the linear method, while the zero output method requires less time spent interacting with current limited conditions.

Humans can perceive differences in sphericity or how spherical an object is, of objects in the hand within a certain error margin. The part of the hand used to distinguish a perfect sphere from a distorted one is examined in this paper. Using geometry, texture, and other physical cues, subjects detect differences in sphericity. In this experiment subjects were asked to discern a perfect steel ball bearing from a distorted one without visual cues. During the final discrimination task, subjects were given a pair of spheres coated in ink, staining their hands where they touched the ball bearings. Photographs were taken of the ink-stained hands and analyzed with a grid based on anatomical features. The sections of the hand most commonly used among all subjects were the thumb and the area between the index and middle finger. These regions had an average ink coverage of between 81 and 100 percent, much higher values than the rest of the hand.

Despite the great diversity of teleoperator designs and applications, their underlying control systems have many similarities. These similarities can be exploited to enable inter-operability between heterogeneous systems. We have developed a network data specification, the Interoperable Telerobotics Protocol, that can be used for Internet based control of a wide range of teleoperators. In this work we test interoperable telerobotics on the global Internet, focusing on the telesurgery application domain. Fourteen globally dispersed telerobotic master and slave systems were connected in thirty trials in one twenty four hour period. Users performed common manipulation tasks to demonstrate effective master-slave operation. With twenty eight (93%) successful, unique connections the results show a high potential for standardizing telerobotic operation. Furthermore, new paradigms for telesurgical operation and training are presented, including a networked surgery trainer and upper-limb exoskeleton control of micro-manipulators.

Force-reflecting teleoperators in which the remote environment is kinesthetically coupled to the operator can considerably increase task performance. Wave-variable-based controllers can support the stable operation of force-reflecting teleoperators under arbitrary communication delays. Transparency in such systems is compromised in order to maintain stability. We had previously proposed a modified wave variable controller that implemented additional wave impedance in the wave variable transformations in order to focus more closely on force tracking. In this paper, we present a new controller for bilateral teleoperators based on the modified wave variable control method which provides superior position and force tracking performance compared to the traditional wave-variable-based method. Moreover, the method has high stability. Theoretical investigation and experimental results confirm the performance of this new controller.

In the area of surgical robotics no standard means of performance evaluation has been established. Thousands of surgeons have gone through the SAGES FLS Program, and the psychomotor skill portion of the program is considered the gold standard in laparoscopic skills evaluation. This research describes the use of the FLS block transfer task to evaluate the performance of both surgeons and non-surgeons teleoperating under different time delay conditions on the University of Washington RAVEN Surgical Robot. Time delays of 0 ms, 250 ms, and 500 ms were used and a statistically significant difference in mean block transfer time as well as mean tool tip path length were shown. For this task no significant difference was shown between the surgeon and non-surgeon groups. Clearly surgeon input and feedback is key to surgical robotic system development, but this result implies that non-surgeon subjects can be tested for simple usability evaluations.

Despite the frequent use of the Lawrence architecture since its introduction in the early 90's, its global transparency characteristics have not yet been fully analyzed. That is the goal of this paper. We state and prove necessary and sufficient conditions for transparency, with special attention to the information sent across the communication layer. In particular, it is shown that transparency can be preserved even though one, and even two, communication channels are set to zero. The results may serve as a guideline for transparent teleoperator design.

In this paper we analyzed the performance of a peer-to-peer haptic collaboration system with two users jointly manipulating an object with mass and damping properties. We used objective measures to compare tuned PD, wave variables and time domain passivity controllers subject to real time delays from the Internet through similar experimental parameters. We set up a packet reflector network at our collaborators' servers in order to able to perform the experiment with subjects located in the same laboratory. Subjects were blinded to which controller was used and received them in a randomized sequence. UDP data packets were used for haptic data communication and the packet transmission rate was maintained at 1000 Hz. Our experimental results show that the tuned PD controller gave the best performance in terms of position error and wave variables in terms of force.

Force-reflecting teleoperators in which the remote environment is kinesthetically coupled to the operator can considerably increase task performance. However, wave-variable-based controllers can support the stable operation of force-reflecting teleoperators under arbitrary communication delays. Transparency in such systems is compromised in order to maintain stability. In this paper, we present a new controller for bilateral teleoperators based on the wave variable control method which provides superior force tracking performance compared to the traditional wave-variable-based method. Additionally, this method also improves the phase delay induced by the latency in the communication network. Both simulation and experimental results confirm the performance of this new controller.

In minimally invasive telesurgical systems, displaying the forces measured at the slave side is an important issue. Latency in communication lines limits the transmission of vivid tactile sensations and drives the system unstable. In this paper, we propose a new model-based approach for the bilateral control of a telesurgical robot using time-domain passivity control. This method consists of a virtual slave model implemented at the master side to estimate the force on the slave side. During the operation, the estimated force from the virtual slave is added to the actual measured force transmitted from the slave in order to maintain the passivity of the system. Both simulation and experimental results confirm the performance of this new controller.

The focus of this special issue is to show how the main achievements on different technical topics relevant to the development of bioinspired and bioapplied mechatronic devices and robotic systems. The 19 articles in this special issue are summarized here.

In order to enable robotic surgery without human assistance, a means must be developed to change tools. As part of the larger Trauma Pod Project, we developed the Tool Rack Subsystem - an automated tool rack capable of holding, accepting, and dispensing up to 14 tools for the da Vinci surgical robot. Borrowing some techniques from industrial automation, we developed a robust system capable of presenting any stored tool in 700ms or less. Tools are positively retained in a sterilizable carousel in a compliant manner designed to accomodate misalignment during tool exchange. RFID equipment is integrated into the system and the tools so that tools can be inventoried and presented by function or serial number instead of rack position. The resulting device has completed testing and integration into the Trauma Pod system and met all its design requirements.

In robot-assisted surgery, surgical tools interact with tissues that have nonlinear mechanical properties. For situations where a pre-specified trajectory of tool positions (or applied forces) is desired, there are many controller designs that might be used. Four candidates are comparatively evaluated here, via computer simulation involving a nonlinear model of soft tissue behavior during grasping actions. The parameters for this model were obtained experimentally (in earlier work). The four candidate controllers are: (1) a well- tuned PID controller; (2) feedback linearization in combination with deadbeat control; (3) an optimal open-loop control law obtained via minimization of a quadratic cost function; and (4) a model predictive controller. Simulation trials are used to compare the transient performance of these candidate controllers under different assumptions regarding input and output noises. The conditions where each of the candidates is best are characterized.

The exoskeleton robot, serving as an assistive device worn by the human (orthotic), functions as a human-amplifier. Setting the human machine interface (HMI) at the neuro-muscular level may lead to seamless integration and an intuitive control of the exoskeleton arm as a natural extension of the human body. At the core of the exoskeleton HMI there is a myoprocessor. It is a model of the human muscle, running in real-time and in parallel to the physiological muscle, that predicts joint torque as a function of the joint kinematics and neural activation levels. The study is focused on developing a myoprocessor based on the Hill phenomenological muscle model. Genetic algorithms were used to optimize model internal parameters using an experimental database that provides inputs to the model and allows for performance assessment. The results indicate high correlation between joint moment predictions of the model and the measured data. Consequently, the myoprocessor seems an adequate model, sufficiently robust for further integration into the exoskeleton control system.

A noisy behavior of the time domain passivity controller during the period of low velocity is analyzed. Main reasons of the noisy behavior are investigated through a simulation with a one-DOF haptic interface model. It is shown that the PO/PC is ineffective in dissipating the produced energy when the sign of the velocity, which is numerically calculated from the measured position, is suddenly changed, and when this velocity is zero. These cases happen during the period of low velocity due to the limited resolution of the position sensor. New methods, ignoring the produced energy from the velocity sign change, and holding the control force while the velocity is zero, are proposed for removing the noisy behavior. The feasibility of the developed methods is proved with both a simulation and a real experiment.

A noisy behavior of the time-domain passivity controller during the period of low velocity is analyzed. Main reasons of the noisy behavior are investigated through a simulation with a one-DOF haptic interface model. It is shown that the PO/PC is ineffective in dissipating the produced energy when the sign of the velocity, which is numerically calculated from the measured position, is suddenly changed, and when this velocity is zero. These cases happen during the period of low velocity due to the limited resolution of the position sensor. New methods, ignoring the produced energy from the velocity sign change, and holding the control force while the velocity is zero, are proposed for removing the noisy behavior. The feasibility of the developed methods is proved with both a simulation and a real experiment.

Advances in surgical technology allow physicians to more effectively provide care to their patients. Minimally invasive surgery (MIS) has revolutionized the way a significant number of procedures are performed. Advances in technology have led to the fusion of MIS techniques and robotic devices; however, such systems are currently large and cumbersome. By optimizing a spherical mechanism based on in-vivo data collected during MIS procedures, this paper focuses on a bottom-up approach in developing a new class of surgical robot arms. The spherical mechanism is a rotational manipulator with all axes intersecting at the center of the sphere. Locating the rotational center of the mechanism at the MIS port makes this class of mechanism a suitable candidate for the first two links of a surgical robot for both minimally invasive and open surgery. For optimizing the mechanism structure, the forward and inverse kinematics, as well as the Jacobian matrix, were derived. Using the Jacobian, mechanism isotropy was considered as the performance metric. The dexterous workspace (DWS) is defined as a high dexterity region defined by a right circular cone with a vertex angel of 60/spl deg/ in which 95% of the tool motions are contained based on in-vivo measurements. The extended dexterous workspace (EDWS) is defined as the workspace required to reach the entire abdominal cavity with MIS instruments and defined by a cone with an elliptical cross section created by two orthogonal vertex angels of 60/spl deg/ and 90/spl deg/. Optimization across both the DWS and a superset of the EDWS led to a mechanism configuration with link length angles of 74/spl deg/ and 60/spl deg/ that maximizes kinematic performance and compactness. The workspace of this design covers the entire EDWS and is the optimal design for the next generation of surgical manipulator. By directly applying in-vivo experimental data from MIS in order to optimize the spherical manipulator a design that maximizes performance and minimizes size has been developed. A pair of prototype manipulators is developed based on these results.

Biological designs offer roboticists a rich source of mechanisms for the challenge of controlling movement. Our device draws upon this resource, modeling the muscle spindle, a biological sensor which transduces muscle length and velocity for kinesthetic awareness and movement control. The three core neural and mechanical elements of the muscle spindle are identified and implemented in precision engineering hardware using performance specifications derived from biological literature. Intrafusal muscle is modeled by a linear actuator fast enough to replicate muscle dynamics. Its step response exhibits 27 ms rise time and 9.2% overshoot. Sensory region transduction of strain into voltage is modeled by a strain-gauged cantilever 51 /spl mu/m thick. A voltage-controlled oscillator, encoding voltage as a frequency-modulated square wave, models action potential frequency encoding. The transducer exhibits the desired linear response with a 34-nm/Hz sensitivity. The three subsystems were combined to perform integrated systems testing. Driving the actuator with simple position control, the device detects trajectory-tracking errors introduced by phase lag and perturbation. Driving the actuator with physiologically based force control, the device successfully replicates the major features of muscle spindle response under ramp and sinusoidal position inputs. Applications include motor control research and novel sensor design for prosthetics and engineering.

A new bilateral control scheme is proposed to ensure stable teleoperation under a wide variety of environments and operating speeds. System stability is analyzed in terms of the time-domain definition of passivity. A previously proposed energy-based method is extended to a 2-port network, and the issues in implementing the "passivity observer" and "passivity controller" to teleoperation systems are studied. The method is tested with our two-degrees-of-freedom master/slave teleoperation system. Stable teleoperation is achieved under conditions such as hard wall contact (stiffness >150 kN/m) and hard surface following.

We propose a new time-domain passivity observer (PO) and passivity controller (PC) which removes the constant-velocity assumption during one sample time, which was used in our previous PO/PC approach. A new sampled-time definition of passivity is introduced, and this new definition is compared with the previous sampled-time definition of passivity. Through this comparison, we propose the more accurate PO/PC approach. The proposed new PO/PC approach is applied to the "Excalibur" haptic interface system with very high stiffness (K = 120 kN/spl middot/m) virtual environment, and stable contact is demonstrated.

A new method to control a flexible manipulator with noncollocated feedback is proposed. We introduce a method to implement the time-domain passivity-control approach to a flexible manipulator with noncollocated feedback, which could not be treated with the previous time-domain passivity-control framework due to a possible active transfer function from the input to the noncollocated output. The proposed method is simulated with a single-link flexible manipulator, and a good control performance is obtained.

We propose new time domain passivity observer (PO) and passivity controller (PC) which removes the constant velocity assumption during one sample time, which was used in our previous PO/PC approach. A new sampled time definition of passivity is introduced, and this new definition is compared with the previous sampled time definition of passivity. Through this comparison, we propose the more accurate PO/PC approach. The proposed new PO/PC approach is applied to "Excalibur" haptic interface system with very high stiffness (K = 120 KN/m) virtual environment (VE), and stable contact is demonstrated.

Handheld devices are enhancing many aspects of our lives. As increasingly complex devices appear with decreasing form factors, haptics may become an essential tool for interacting with them. In this regime of operation, issues of power, weight and volume are of significant importance. The haptic thresholds of the index finger for active exploration of a two dimensional virtual environment for two icon alignments and two finger motions were measured. Using all possible combinations of two finger motions, flexion/extension and finger abduction/adduction, and two icon alignments, vertical and horizontal, were measured separately. Haptic thresholds ranged from 15 to 24 milliNewtons. Thresholds were affected by finger motion, but not by icon alignment.

In this paper a modification of the time domain passivity controller is presented to improve its performance and transparency in case of multi degrees of freedom (dof) haptic interaction. In multi-dof application the concept needs to be extended by additional conditions to distribute the adaptive damping appropriately among the degrees of freedom. This can be solved by using the geometrical information coded in the output signals of the system. Experiments show the validity of this concept.

A recently proposed method for stabilizing haptic interfaces and teleoperation systems was tested with a "PHANToM" commercial haptic device. The "passivity observer" (PO) and "passivity controller" (PC) stabilization method was formed to stabilize the system but also excite high frequency mode in the device. To solve this problem, we propose a method to use a time-varying desired energy threshold instead of fixed zero energy threshold for the PO, and make the actual energy input follow the time-varying energy threshold. With the time-varying energy threshold, we make the PC control action smooth without sudden impulsive behavior by distributing the dissipation. The proposed new PO/PC approach is applied to PHANToM with high stiffness (K=500 N/m), and stable and smooth contact is guarantee. Resetting and active environment display problems can also be solved with the reference energy following idea.

In this survey paper, the authors analyze the general structure of robotic systems for computer-assisted surgery, present a classification of such systems based on the degree of "intelligence" of the tools, and discuss some examples of different classes of devices. Computer-assisted surgery accelerated progress is related, on the one hand, to the improvement of medical imaging techniques and, on the other hand, to the evolution of surgical instrumentation. The integration of these two factors has determined an extraordinary progress that is not just a "linear" temporal development, but it is a "discontinuity" as regards traditional surgical procedures. Specifically, the authors consider the following classes of robotic-derived surgical devices/systems: a) handheld tools augmenting the capabilities of the surgeon; b) teleoperated surgical tools; and c) autonomous surgical robots. The paper will focus essentially on the analysis of systems and components of robots and tools designed for minimally invasive surgery. Although different classification methods exist on the basis of the clinical needs and/or on the design approach, the devices which will be illustrated in this paper are classified on the basis of their scale, degrees of freedom, autonomy, embedded intelligence, and features of the interface between the surgeon and the patient.

Minimally invasive surgery involves a multidimensional series of tasks requiring a synthesis between visual information and the kinematics and dynamics of the surgical tools. Analysis of these sources of information is a key step in mastering MIS, but may also be used to define objective criteria for characterizing surgical performance. The BlueDRAGON is a new system for acquiring the kinematics and dynamics of two endoscopic tools synchronized with the visual view of the surgical scene. It includes two four-bar passive mechanisms equipped with position and force torque sensors for measuring the positions and orientations of the two endoscopic tools along with the forces and torques (F/T) applied by the surgeon's hands. The methodology of decomposing the surgical task is based on a fully connected, 28 finite-states Markov model where each states corresponded to a fundamental tool/tissue interaction based on the tool kinematics and associated with unique F/T signatures. The experimental protocol includes seven MIS tasks performed on an animal model by 30 surgeons at different levels of their residency training including expert surgeons. From the preliminary analysis of these data, the major differences between residents at different skill levels are discussed. Systems like surgical robots or virtual reality simulators that inherently measure the kinematics and dynamics of the surgical tool may benefit from inclusion of the proposed methodology for the analysis of efficacy and objective evaluation of surgical skills during training.

A new bilateral control scheme is proposed to ensure stable teleoperation under a wide variety of operating conditions. To guarantee the stability of a teleoperation system, a previously proposed energy-based method is extended to a two-port network. The issues in implementing the "passivity observer" and "passivity controller" to teleoperation systems are studied. The method is tested with our two-DOF master/slave teleoperation system. Totally stable teleoperation is achieved under conditions such as hard wall contact (stiffness>150 kN/m) and hard surface following.

A new, energy-based method is proposed for guaranteeing the stability of large classes of control systems with minimum performance losses. Based on a network presentation, the large classes of control systems are analyzed in a unified framework. In this unified network model, the concept of passivity is used to study the stability of large classes of control systems. For guaranteeing the stability condition, the time-domain passivity controller is extended to a 2-port network to make the controller 2-port passive. The developed method is tested with numerical simulation in the regulation of a single link flexible manipulator. Totally stable control is achieved under a wide variety of operating conditions and uncertainties without any model information.

A patent-pending, energy-based method is presented for controlling a haptic interface system to ensure stable contact under a wide variety of operating conditions. System stability is analyzed in terms of the time-domain definition of passivity. We define a "passivity observer" (PO) which measures energy flow in and out of one or more subsystems in real-time software. Active behavior is indicated by a negative value of the PO at any time. We also define the "passivity controller" (PC), an adaptive dissipative element which, at each time sample, absorbs exactly the net energy output (if any) measured by the PO. The method is tested with simulation and implementation in the Excalibur haptic interface system. Totally stable operation was achieved under conditions such as stiffness >100 N/mm or time delays of 15 ms. The PO/PC method requires very little additional computation and does not require a dynamical model to be identified.

An energy-based method is presented for controlling a haptic interface system to ensure stable contact under a wide variety of operating conditions. System stability is analyzed in terms of the time-domain definition of passivity. We define a "passivity observer" (PO) which measures energy flow in and out of one or more subsystems in real-time software. Active behavior is indicated by a negative value of the PO at any time. We also define the "passivity controller" (PC), an adaptive dissipative element which, at each time sample, absorbs exactly the net energy output (if any) measured by the PO. The method is tested with simulation and implementation in the "Excalibur" haptic interface system. Totally stable operation was achieved under conditions such as stiffness >100 N/mm or time delays of 15 ms. The PO/PC method requires very little additional computation and does not require a dynamical model to be identified.

In this paper, two major practical issues are studied to improve the performance of a new energy based method of achieving stable, high performance haptic interface control. The first issue is related to resetting the amount of energy accumulated in the passivity observer for faster operation. A heuristic method is derived and experimentally tested for the resetting and it is shown to help the PC to operate sooner when the system gets active. The second one was noise in velocity measurements being magnified into audible force signals by the controller. This issue was addressed by the introduction of a velocity threshold and it is verified that the velocity threshold makes the PC more free from the noise effect at low velocity. Experimental results are presented for the "Excalibur" haptic device.

Creating a compelling haptic sense of immersion in a virtual environment is a challenging task for the control engineer. A haptic display must render both low impedance free-space motion and high impedance rigid constraints while ensuring stable interaction. This paper outlines a control design approach for the most common haptic display implementation, the impedance display. Two-port absolute stability criteria are used to develop explicit design bounds for virtual coupling networks which guarantee system stability for a broad class of human operators and virtual environments. The technique is applied to the Excalibur three-axis force display. The resulting absolutely stable haptic interface is the centerpiece of a virtual building block simulation which emulates the behavior of LEGO/sup TM/ bricks in a virtual environment.

This paper presents a biomimetic sensor for transducing displacements. Our sensor is a robotic analog of the biological muscle spindle, an actuated position sensor which transduces muscle displacement for kinaesthetic awareness. The mechanical filter exhibits the desired step response. The transducer possesses the desired linear response with a sensitivity of 34nm/Hz. Finally, the encoder circuitry successfully maps the millivolt output to a pulse frequency range of 1150 Hz to 12.5 kHz. Results from an integrated system test show that the sensor can successfully detect errors in trajectory tracking introduced by both phase lag and perturbations. By physically realizing the hypothesized core features of a biological muscle spindle in engineering hardware, we evoked the type of actuated sensor output seen in the biological muscle spindle, a widely utilized tool of biological motor control.

This paper describes a novel 4-axis microgripping system consisting of two fingers, each driven by a 2-axis moving coil actuator taken from a CD-lens assembly. These electromagnetic actuators are small, very linear, virtually frictionless and low cost. We measured the electrical actuator parameters and characterized the actuator performance in terms of displacement vs. current, force vs. current and resonant frequency. Experimental results indicate that the proposed microgripping system can be an attractive solution to the problem of micromanipulating small objects for precision manufacturing and biotechnology with high accuracy in a relatively large workspace.

This paper addresses fundamental stability and performance issues associated with haptic interaction. It generalizes and extends the concept of a virtual coupling network, an artificial link between the haptic display and a virtual world, to include both the impedance and admittance models of haptic interaction. A benchmark example exposes an important duality between these two cases. Linear circuit theory is used to develop necessary and sufficient conditions for the stability of a haptic simulation, assuming the human operator and virtual environment are passive. These equations lead to an explicit design procedure for virtual coupling networks which give maximum performance while guaranteeing stability. By decoupling the haptic display control problem from the design of virtual environments, the use of a virtual coupling network frees the developer of haptic-enabled virtual reality models from issues of mechanical stability.

A method is described to characterize and experimentally measure the dynamic performance of haptic display devices. The method characterizes the response to impulse inputs of various frequencies characteristic of simulating hard contacts in virtual environments. By comparing the experimentally measured velocity just after the impulse with the actual velocity, a dimensionless measure of structural distortion is derived. The method is easy to apply because no additional sensors or test fixtures are required. This paper presents a derivation of the structural deformation ratio for the single degree of freedom case, generalization to N-DOF spatial devices, and experimental results for a single axis of a rugged haptic device in our laboratory.

A haptic interface is a kinesthetic link between a human operator and a virtual environment. This paper addresses stability and performance issues associated with haptic interaction. It generalizes and extends the concept of a virtual coupling network, an artificial connection between a haptic display and a virtual world, to include both the impedance and admittance models of haptic interaction. A benchmark example exposes an important duality between these two cases. Linear circuit theory is used to develop necessary and sufficient conditions for the stability of a haptic simulation, assuming the human operator and virtual environment are passive. These equations lead to an explicit design procedure for virtual coupling networks which give maximum performance while guaranteeing stability. By decoupling the haptic display control problem from the design of virtual environments, the use of a virtual coupling network frees the developer of haptic-enabled virtual reality models from issues of mechanical stability.

The McKibben artificial muscle is a pneumatic actuator whose properties include a very high force to weight ratio. This characteristic makes it very attractive for a wide range of applications such as mobile robots and prosthetic appliances for the disabled. Typical applications often require a significant number of repeated contractions and extensions or cycles of the actuator. This repeated action leads to fatigue and failure of the actuator, yielding a life span that is often shorter than its more common robotic counterparts such as electric motors or pneumatic cylinders. In this paper, we develop a model that predicts the maximum number of life cycles of the actuator based on available uniaxial tensile properties of the actuator's inner bladder. Experimental results, which validate the model, reveal McKibben actuators fabricated with natural latex rubber bladders have a fatigue limit 24 times greater than actuators fabricated with synthetic silicone rubber at large contraction ratios.

Kinematic conditioning of robot manipulators is the problem where small motions in Cartesian space cause excessive joint velocities. This problem is significant in teleoperation. Haptic feedback provides the bi-directional flow of information which allows the operator to control the telerobot interactively. Haptic feedback of kinematic conditioning is proposed as a new approach to achieve better performance in telerobotic control near kinematic singularities. Four different singularity force feedback methods are defined and studied. Experimental results with a force feedback master and simulated slave system show that teleoperation performance near singular configurations was affected and improved by using singularity force feedback.

A combined university/industry team has developed a prototype system for handling protein crystals aboard the space station. This system uses a miniature direct drive robot, CCD television cameras, and a client-server computing system using Internet protocols to support the capture of protein crystals from aqueous growth solutions. The system was demonstrated between Huntsville AL. and Seattle WA. An operator in Huntsville controlled the mini robot by invoking predefined relative and absolute macro files. The operators observed results using video images sent through the Internet link using Cu-SeeMe video conferencing software. In 3 of 4 trials, the operators successfully captured 0.5 mm simulated protein crystals into a glass capillary. The system is a promising start for the development of a space-station based remote protein crystal analysis facility.

This paper reports mechanical testing the modeling results for the McKibben artificial muscle pneumatic actuator. This device contains an expanding tube surrounded by braided cords. We report static and dynamic length-tension testing results and derive a linearized model of these properties for three different models. The results are briefly compared with human muscle properties to evaluate the suitability of McKibben actuators for human muscle emulation in biologically based robot arms.

We present a general approach to approximating positive time-frequency distributions (TFDs) through nonlinear combinations of spectrograms. Closed-form solutions for the combinations are obtained via optimization of entropy functionals subject to an energy constraint. We apply two such combinations to generating approximate TFDs for whale sounds and speech. Through these applications, it can be seen that these methods give results superior to that achieved with individual spectrograms, and remarkably close to the positive TFDs obtained via computationally-intensive methods.

A method is described and validated for the automatic calibration of analog sine-wave quadrature sensors, such as optical encoders, embedded in a functioning system. The algorithm uses a Kalman filter to estimate the true position of the direct-drive actuator joint using a model of it's dynamics, an applied actuator command and measurements from the uncalibrated sensor. From the estimated true position, a lookup table is constructed which corrects sensor errors. Our results indicate that this method achieves accuracies typical of interferometric calibration, without requiring an external measurement device. The accuracy is surprisingly robust to modeling errors.

This paper reports mechanical testing and modeling results for the McKibben artificial muscle pneumatic actuator. This device first developed in the 1950's, contains an expanding tube surrounded by braided cords. The authors report static and dynamic length-tension testing results and derive a linearized model of these properties for three different models. The results are briefly compared with human muscle properties to evaluate the suitability of McKibben actuators for human muscle emulation in biologically based robot arms.>

A previously developed 3 axis mini direct drive robot has been enhanced with two additional direct drive axes for general positioning and orientation of an axially symmetric tool. The arm has a work volume of about 50 cc and will have 5-10 micron or better resolution and repeatability. The arm forms an initial prototype for the NASA/University of Washington MicroTrex flight telerobotics experiment. The contemplated terrestrial applications include handling sub-microliter liquid samples for electrophoresis, and micro-manipulation with scaled force reflection.>

We demonstrate that two previously proposed methods for combining the information content from multiple spectrograms into a single, positive time-frequency function are optimal in a cross-entropy sense. The goal in combining the spectrograms is to obtain an improved approximation of the joint time-frequency signal density by overcoming limitations of any single spectrogram. An example of each method is provided, and results are compared with spectrograms and a Cohen-Posch (1985) time-frequency density (TFD) of a nonstationary pulsed tone signal. The proposed combinations are effective and can be efficiently computed.>

Shared compliant control has been incorporated into an advanced six-degree-of-freedom force-reflecting telemanipulation system. With this system the authors have investigated the effect of time delay on human telemanipulation task performance. Time delays of between 2 and 4096 ms were introduced between master and slave arms, and high-precision peg-in-hole tasks were performed by six test operators with two modes of control: kinesthetic force feedback (KFF) and shared compliant control (SCC). Task performance was quantified in terms of the completion time (CT) and the sum of square forces (SOSF). The experimental results demonstrate the superiority of SCC over KFF for time-delayed telemanipulation. SCC has significantly lower rates of increase than KFF in both CT and SOSF with time delay. Only SCC enabled task performance at delays above 1 s. Constant force maintenance tasks were also performed to investigate the effect of time delay on the stability of force reflection. SCC also has beneficial effects on telemanipulation without time delay.>

A method is presented for using hidden Markov models (HMMs) for the analysis of force, torque, and position signals from sensors in manufacturing machines. The HMM can detect the transitions between contact states and compute a measure of the task quality using a model of the task developed by the manufacturing engineer and optimized on training data. The HMM method has been evaluated in extensive experimentation with teleoperation and the results suggest even higher effectiveness in automation and manufacturing applications.>

Recent work in real-time distributed computation and control has culminated in a prototype force-reflecting telemanipulation system having dissimilar master (cable-driven force-reflecting hand controller) and slave (PUMA 560 robot with custom controller), extremely high sampling rate (1000 Hz), and low loop computation delay (5 ms). In a series of experiments with this system and five trained test operators covering more than 100 h of teleoperation, performance in a series of generic and application-driven tasks with and without force feedback was measured, and with control shared between teleoperation and local sensor referenced control. Measurements defining task performance include 100-Hz recording of six-axis force-torque information, task completion time, and visual observation of predefined task errors. It is shown that all performance measures improved as capability was added along a spectrum of capabilities ranging from pure position control through force-reflecting teleoperation and shared control. Performance was maximal for the barehanded operator.>

Kinesthetic force feedback provides measurable increase in remote manipulation system performance. Intensive computation time requirements or operation under conditions of time delay can cause serious stability problems in control-system design. The author presents a simplified linear analysis of this stability problem for the forward-flow generalized architecture and uses the hybrid two-port representation to express the loop gain of the traditional master-slave architecture, which can be subjected to similar analysis. The hybrid two-port representation is also used to express the effects on the fidelity of manipulation or feel of one design approach used to stabilize the forward-flow architecture. The results suggest that, when local force feedback at the slave side is used to reduce manipulator stability problems, a price is paid in terms of telemanipulation fidelity.>

Time delays of between 2 ms and 4096 ms were introduced between master and slave manipulators in advanced 6-DOF (degrees-of-freedom) force-reflecting telemanipulation system. The effects of the imposed delay were quantified in terms of completion time (CT) and sum-of-squared-force (SOSF) with six test operators and two modes of control: kinesthetic force feedback (KFF) and shared compliant control (SCC). KFF applies forces and torques proportional to those sensed by the slave robot to the human operator through the master. SCC uses the force/torque information entirely at the slave side to implement a variation on impedance control in parallel with the operator's position commands. Experiments measured effects of human operator grasp variation on stability of contact with KFF. CT and SOSF performance in a peg-in-hole task decreased linearly with imposed time delay using KFF, and decreased at a lower rate with SCC. SCC enabled task performance for 2 and 4 second delays, critical values for low-earth-orbit applications.>

The application of a hybrid two-port model to teleoperators with force and velocity sensing at the master and slave is presented. The interfaces between human operator and master, and between environment and slave, are ports through which the teleoperator is designed to exchange energy between the operator and the environment. By computing or measuring the input-output properties of this two-port network, the hybrid two-port model of an actual or simulated teleoperator system can be obtained. It is shown that the hybrid model (as opposed to other two-port forms) leads to an intuitive representation of ideal teleoperator performance and applies to several teleoperator architectures. Thus measured values of the h matrix or values computed from a simulation can be used to compare performance with the ideal. The frequency-dependent h matrix is computed from a detailed SPICE model of an actual system, and the method is applied to a proposed architecture.>

Experiments and simulations of a single-axis force-reflecting teleoperation system have been conducted to investigate the problem of contacting a hard environment and maintaining a controlled force in teleoperation in which position is fed forward from the hand controller (master) to the manipulator (slave), and force is fed back to the human operator through motors in the master. The simulations, using an electrical circuit model, reproduce the behavior of the real system, including effects of human operator biomechanics. It is shown that human operator properties, which vary as a result of different types of grasp of the handle, affect the stability of the system in the hard-contact task. The effect of a heavier grasp on the handle is equivalent to increased hand-controlled velocity damping in terms of the systems stability in the contact task, but control system damping sufficient to guarantee stable contact results in perceptible sluggishness of the control handle's response in free motion. These results suggest that human operator biomechanics must be taken into account to guarantee stable and ergonomic performance of advanced teleoperators.>

A new device for human operator control of a robotic gripper has been developed and preliminary evaluation has been performed. The JPL Force Reflecting Hand Trigger system features: an instrumented index finger trigger with load cell detection of finger force. A servo controlled, lead screw driven backdrive capability by which the trigger's position can be made to follow that of the remotely controlled gripper. And a novel feedback mechanism by which clamping force or some other signal can be fed back via a swiveling motion, also servo controlled, of the trigger surface (force reflection). This system has undergone preliminary testing in which the amount of force reflection is varied and dynamic force tracking response is observed.

An experimental telerobotics (TR) simulation is described suitable for studying human operator (HO) performance. Simple manipulator pick-and-place and tracking tasks allowed quantitative comparison of a number of calligraphic display viewing conditions. An enhanced perspective display was effective with a reference line from target to base, with or without a complex three-dimensional grid framing the view. This was true especially if geometrical display parameters such as azimuth (AZ) and elevation (EL) were arranged to be near optimal. Quantitative comparisons were made possible utilizing control performance measures such as root mean square error (rmse). There was a distinct preference for controlling the manipulator in end-effector Cartesian space for our primitive pick-and-place task, rather than controlling joint angles and then, via direct kinematics, the end-effector position. An introduced communication delay was found to produce decrease in performance. In considerable part, this difficulty could be compensated for by preview control information. That neurological control of normal human movement contains a sampled data period of 0.2 s may relate to this robustness of HO control to delay.

Optimal presentation of three-dimensional information on a two-dimensional display screen requires careful design of the projection to the display surface. Monoscopic perspective projection alone is usually not sufficient to represent three-dimensional spatial information. It can, however, be improved by the adjustment of perspective parameters and by geometric visual enhancements such as reference lines and a background grid. Stereoscopic display is another method of providing three-dimensional information to the human operator. Two experiments are performed with three-axis manual tracking tasks. The first experiment investigates the effects of perspective parameters on tracking performance. The second experiment investigates the effects of visual enhancements for both monoscopic and stereoscopic displays. Results indicate that, though stereoscopic displays do generally permit superior tracking performance, monoscopic displays can allow equivalent performance when they are defined with optimal perspective parameters and provided with adequate visual enhancements.