This paper presents the architectural details of an evolving open source/open architecture software framework for developing organ-level surgical simulations. Our goal is to facilitate shared development of reusable models, to accommodate heterogeneous models of computation, and to provide a framework for interfacing multiple heterogeneous models. The framework provides an application programming interface for interfacing dynamic models defined over spatial domains. It is specifically designed to be independent of the specifics of the modeling methods used, and therefore facilitates seamless integration of heterogeneous models and processes. Furthermore, each model has separate geometries for visualization, simulation, and interfacing, allowing the model developer to choose the most natural geometric representation for each case. Input/output interfaces for visualization and haptics for real-time interactive applications have also been provided.

Published in: IEEE Proceedings. VR 2005. Virtual Reality, 2005

An important area of research in the teleoperation literature is to develop systematic methods to quantitatively compare different manipulator designs in application critical tasks. Such quantitative methods are especially important during design of the manipulators to make an informed decision among various design alternatives. In this paper, a novel method to quantitatively compare different sensory schemes for a teleoperation system is introduced. This method evaluates the sensory schemes by comparing the norm of the a posteriori error covariance matrices of the Kalman filters for each configuration. The main advantage of this method is that it allows to quantitatively compare arbitrary sensory configurations.

In this letter, teleoperation controller design for haptic exploration and telemanipulation of soft environments is studied. First, a new measure for fidelity in teleoperation is introduced which quantifies the teleoperation system's ability to transmit changes in the compliance of the environment. This sensitivity function is appropriate for the application of telesurgery, where the ability to distinguish small changes in tissue compliance is essential for tasks such as detection of embedded vessels. The bilateral teleoperation controller design problem is then formulated in a task-based optimization framework as the optimization of this metric, with constraints on free-space tracking and robust stability of the system under environment and human operator uncertainties. The control design procedure is illustrated with a case study. The analysis is also used to evaluate the effectiveness of using a force sensor in a teleoperation system.

Previous research on teleoperation has focused on manipulation of hard objects. However, the design constraints are different in applications that involve manipulation of deformable objects, such as robotic telesurgery. In this paper a new measure for fidelity in teleoperation is introduced which quantifies the teleoperation system's ability to transmit changes in the compliance of the environment. This sensitivity function is highly appropriate for the application of telesurgery, where the ability to distinguish small changes in tissue compliance is essential for tasks such as tumor detection. The bilateral teleoperation controller design problem is then formulated as the optimization of this new metric with constraints on free space tracking requirements and robust stability of the system under environment and human operator uncertainties. The robust stability analysis can be applied to any teleoperator plant and guarantee stability given an uncertainty model. The analysis is also extended to evaluate effectiveness of using a force sensor in the teleoperation system.

An important missing piece in the medical robotics literature is the lack of systematic methods to quantitatively compare different manipulator designs, and to evaluate kinematic configurations chosen for telesurgical manipulators in application-critical tasks. Such a quantitative method is especially important during design stage to make an informed decision between various design alternatives. In the paper, a quantitative method to evaluate the kinematic ability of surgical manipulators to perform the critical tasks of suturing and knot tying is presented. The proposed method does not require a physical prototype. This is achieved by running typical tool motions during these tasks through the inverse kinematics of the manipulators and checking if the system can accommodate the desired motions. The system can perform a given motion if the whole trajectory lies continuously within the workspace of the manipulator. Open surgical suturing motion data collected from experiments done with expert surgeons is used as the set of desired tool motions used in the analysis. The method is applied to compare two different wrist configurations of telesurgical slave manipulators, intended for use in minimally invasive surgery, by looking at the requirements on joint ranges and wrist manipulability during these motions.

Realistic behavior of deformable objects is essential for many applications such as simulation for surgical training. Existing techniques of deformable modeling for real time simulation have either used approximate methods that are not physically accurate or linear methods that do not produce reasonable global behavior. Nonlinear finite element methods (FEM) are globally accurate, but conventional FEM is not real time. In this paper, we apply nonlinear FEM using mass lumping to produce a diagonal mass matrix that allows real time computation. Adaptive meshing is necessary to provide sufficient detail where required while minimizing unnecessary computation. We propose a scheme for mesh adaptation based on an extension of the progressive mesh concept, which we call dynamic progressive meshes.

Haptic interaction is an increasingly common form of interaction in virtual environment simulations. This medium introduces some new challenges. In this paper we study the problem arising from the difference between the sampling rate requirements of haptic interfaces and the significantly lower update rates of the physical models being manipulated. We propose a multirate simulation approach which uses a local linear approximation. The treatment includes a detailed analysis and experimental verification of the approach. The proposed method is also shown to improve the stability of the haptic interaction.

In this paper, various aspects of robotic telesurgery are studied. After a general introduction to laparoscopic surgery and medical applications of robotics, the UC Berkeley/Endorobotics Inc./UC San Francisco Telesurgical Workstation, a master-slave telerobotic system for laparoscopic surgery, is introduced, followed by its kinematic analysis, control, and experimental results. Some conceptual and future issues on telesurgery are discussed, including teleoperation and hybrid control, focusing on the special requirements of telesurgery.

A hierarchical telerobotic control system for experiments with model-based vision feedback is described. The human operator issues commands to a high-level controller which plans actions to be carried out by the low-level controller. A feedforward model of the robot's motion is used by the vision system to look for visual enhancements on the robot in their anticipated positions. A middle-level module is planned which will alter low-level control gains to maintain performance within error bounds. Results of experiments with the vision system and studies of control strategies for vision feedback are discussed.>

It is argued that autonomous robotics is not yet feasible, and that instantaneous human control may be difficult because of communication delay and expensive because of labor costs. Therefore, it is argued, human supervisory control with cooperation between man and a partially autonomous distant robot is the most viable engineering solution. Model control of image processing enables efficient visual displays. Cooperative control in paired manipulators required rapid controller signal information to be passed over a cross-filter in a heuristic study.>

Position and rate control are the two common manual control modes in teleoperations. Human operator performance using the two modes is evaluated and compared. Simulated three-axis pick-and-place operations are used as the primary task for evaluation. First, ideal position and rate control are compared by considering several factors, such as joystick gain, joystick type, display mode, task, and manipulator work space size. Then the effects of the manipulator system dynamics are investigated by varying the natural frequency and speed limit. Experimental results show that ideal position control is superior to ideal rate control, regardless of joystick type or display mode, when the manipulation work space is small or comparable to the human operator's control space. Results also show that when the manipulator system is slow, the superiority of position control disappears. Position control is recommended for small-work-space telemanipulation tasks, while rate control is recommended for slow wide-work-space telemanipulation tasks.

A visual display system serves as an important man-machine interface for efficient teleoperations. However, careful consideration is necessary to display three-dimensional information on a two-dimensional screen effectively. A teleoperation simulator is constructed with a vector display system, joysticks, and a simulated cylindrical manipulator to evaluate various display conditions quantitatively. Pick-and-place tasks are performed, and mean completion times are used as a performance measure. Two experiments are performed. First, effects of variation of perspective parameters on human operator's pick-and-place performance with monoscopic perspective display are investigated. Then visual enhancements of monoscopic perspective display by adding a grid and reference lines are investigated and compared with visual enhancements of stereoscopic display. The results indicate that stereoscopic display does generally permit superior pick-and-place performance, while monoscopic display can allow equivalent performance when it is defined with appropriate perspective parameter values and provided with adequate visual enhancements. Mean completion time results of pick-and-place experiments for various display conditions shown are observed to be quite similar to the normalized rms error results of mannal tracking experiments reported previously.

An experimental telerobotics (TR) simulation is described suitable for studying human operator (HO) performance. Simple manipulator pick-and-place and tracking tasks allowed quantitative comparison of a number of calligraphic display viewing conditions. An enhanced perspective display was effective with a reference line from target to base, with or without a complex three-dimensional grid framing the view. This was true especially if geometrical display parameters such as azimuth (AZ) and elevation (EL) were arranged to be near optimal. Quantitative comparisons were made possible utilizing control performance measures such as root mean square error (rmse). There was a distinct preference for controlling the manipulator in end-effector Cartesian space for our primitive pick-and-place task, rather than controlling joint angles and then, via direct kinematics, the end-effector position. An introduced communication delay was found to produce decrease in performance. In considerable part, this difficulty could be compensated for by preview control information. That neurological control of normal human movement contains a sampled data period of 0.2 s may relate to this robustness of HO control to delay.