Minimally invasive robotic surgery systems are usually controlled by input devices, that are mechanically linked to the environment. These input devices often have a limited workspace, which makes intuitive operation difficult. This paper presents a tracking algorithm of a handheld input device, which combines inertial and optical measurements to obtain accurate and robust state estimates with high update rates and low latency. It is based on the fusion of inertial and optical data in an error state extended Kalman filter. To achieve a high degree of robustness with respect to partial device occlusions, active optical markers are tracked and their 2D positions in the camera planes are directly forwarded to the fusion process. The algorithm can handle partial occlusions of the device in one or all of the cameras. A quality measure is defined, which indicates if tracking performance is sufficient to control a robot. An exemplary task in a medical robotics context verifies the assumption that the tracking system can be used for real-time robot control despite frequent marker occlusions.

This paper presents a disturbance observer for the sigma.7 haptic device. External torques on the haptic device links are observed to distinguish between intended user interaction and accidental collisions of the links. Collisions can be unintentional contact of the human operator, e.g. with his knee, other people accidentally getting in contact with the haptic device or objects falling on it. The observer is based on the dynamic model of the device and sensor measurements from the joint's position sensors and an integrated force/torque sensor. Is is implemented concurrently with a control algorithm that uses the force/torque sensor to effectively reduce the inertia and friction of the sigma.7. Furthermore, forces and torques applied on the grasping unit by the human operator are observed. Experiments are done with the sigma.7 customized for the German Aerospace Center (DLR) by Force Dimension.

This paper presents a high fidelity force feedback teleoperation control for surgical applications. Advanced control methods, such as flexible joint tracking control and passivity observation, are introduced in the direct force reflecting control architecture. A full state feedback controller of the flexible joint slave robot controls the motor position, velocity, the joint torque, and the torque derivative. The pose of the haptic device and the first three derivatives are observed to generate reference states for the robot control using the robot's inverse dynamics model. Interaction forces of the slave and the environment are measured with a force/torque sensor and directly sent back to the master device. Stability is guaranteed with a passivity observer that monitors the energy in the teleoperation system online and disconnects master and slave if the system operates beyond its stable region. The proposed control architecture is implemented with the sigma.7 haptic device and the MIRO robot. It is experimentally shown, that appropriately considering elasticities with full state reference and control of the slave, increases the dynamic range of the system enabling transparent and stable interaction with hard and soft environments.

This paper presents an optical-inertial tracking algorithm with explicit and assured optical marker identification. Active optical markers are sequentially or simultaneously triggered to achieve a maximum in quantity and quality of measurements available for tracking. Markers that appear in a camera image are identified by individual activation and are locally tracked in the 2D-images after initial identification. The 2D-position measurements of the cameras are combined with low latency measurements of acceleration and angular velocity from an inertial measurement unit. An Extended Kalman Filter is used for an ultra-tightly coupled data fusion, that takes advantage of all marker measurements with verified identity. The accurate, low latency tracking is robust with respect to temporary marker occlusions, as needed in applications where a robot is directly controlled with a tracked device. The tracking algorithms are implemented in real-time and verified with a test bed in a medical robotics context.

This paper introduces an ultra-tightly coupled approach to data fusion of optical and inertial measurements. The two redundant sensor systems complement each other well, with the cameras providing absolute positions and the inertial measurements giving low latency information of derivatives. The targeted application is the tracking of handheld input devices for robotic surgery, where landmarks are not always visible to all cameras. Especially when bi-manual operation is considered, where one hand can move between the other hand and a camera, occlusions occur frequently. The ultra-tighly coupled data fusion uses 2D-camera measurements to correct pose estimations in an extended Kalman filter without an explicit 3D-reconstruction. Therefore marker measurements are used to support the pose estimation, even if the marker is only visible in one camera. Experiments were done with an inertial measurement unit and rectified stereo cameras that show the advantage of the approach for the application.

This paper presents the design and control of the sigma.7 haptic device and the new surgical console of the MiroSurge robotic system. The console and the haptic devices are designed with respect to requirements in minimally invasive robotic surgery. Dedicated left and right handed devices are integrated in an operator console in an ergonomic configuration. The height of the whole console is adjustable, allowing the surgeon seated and standed operation. Each of the devices is fully actuated in seven degrees of freedom (DoF). A parallel mechanism with 3 DoF actuates the translational motion and an attached wrist with 3 intersecting axis drives the rotations of the grasping unit. This advantageous design leads to inherently decoupled kinematics and dynamics. Cartesian forces are 20 N within the translational workspace, which is a sphere of about 120 mm diameter for each device. The rotational wrist of the device covers the whole workspace of the human hand and provides maximum torques of about 0.4 Nm. The grasping unit can display forces up to 8 N. An integrated force/torque sensor is used to increase the transparency of the devices by reducing inertia and friction. It is theoretically shown that the non-linear closed loop system behaves like a passive system and experimental results validate the approach. The sigma.7 haptic devices are designed by Force Dimension in cooperation with the German Aerospace Center (DLR). DLR designed the surgical console and integrated the haptic devices in the MiroSurge system.

This paper proposes a method for accurate robotic motion compensation of a freely moving target object. This approaches a typical problem in medical scenarios, where a robotic system needs to compensate physiological movements of a target region related to the patient. An optical tracking system measures the poses of the robot's end-effector and the moving target. The task is to track the target with the robot in a desired relative pose. Arbitrary motion in 6 DoF is covered. The position controller of the medical light-weight robot MIRO is enhanced by a Cartesian displacement observer. The proposed observer feedback preserves the dynamics of the robot, while achieving high accuracy in task space. The target object is equipped with an inertial measurement unit in addition to tracking markers. Target sensor data is fused by an extended Kalman filter in a tightly coupled approach. The robot control and the target tracking in the task space aim to combine accuracy, dynamic performance and robustness to marker occlusions. The algorithms are verified with the DLR MIRO, an experimental target platform, and a commercial tracking system. The experiments demonstrate rapid convergence to desired Cartesian poses and good dynamic tracking performance even at higher target motion speed.

This paper gives an overview of a multimodal training platform developed for minimally invasive robotic surgery, based on the DLR MiroSurge system. It describes the technological components and integration of the hardware and software platform and presents the first integrated training tasks that enable surgeons to get familiar with the robotic system. The training platform shares the same surgical operator workstation as MiroSurge and simulates the behaviour of the telemanipulator arms and the surgical instruments. Like the real system the training platform provides haptic feedback and 3D-vision. However instead of the real telemanipulator itself, a virtual environment with abstracted tasks is connected to the operator workstation. This allows reduction in costs, to provide various levels of difficulty, and to focus on the skills to be taught. Thus a training platform is presented that aims at training a surgeon's skills in handling the robotic system MiroSurge rather than training surgery in general.

This video presents the in-house developed DLR MiroSurge robotic system for surgery. As shown, the system is suitable for both minimally invasive and open surgery. Essential part of the system is the MIRO robot: The soft robotics feature enables intuitive interaction with the robot.

This paper introduces the planning and control software of a teleoperation system for research in minimally invasive robotic surgery. It addresses the problem of how to organize a complex system with 41 degrees of freedom as a flexible configurable platform. Robot setup planning, force feedback control and nullspace handling with three robotic arms are considered. The planning software is separated into sequentially executed planning and registration procedures. An optimal setup is first planned in virtual reality and then adapted to variations in the operating room. The real time control system is structured in hierarchical layers. Functions are arranged in the layers with respect to their domain and maximum response time. The design is flexible and expandable while performance is maintained. Structure, functionality and implementation of planning and control are described. The prototypic robotic system provides intuitive bimanual bilateral teleoperation within the planned working space.

In this paper a sensor fusion for pose estimation using optical and inertial data is presented. The proposed algorithm is based on extended Kalman filtering and fuses data from an optical tracking system and an inertial measurement unit. These two redundant sensor systems complement each other well, with the tracking system providing absolute positions and the inertial measurements giving low latency information of derivatives. Models for both sensors are given respecting the different sampling times and latencies. Another key issue is to use information about every landmark, i.e. marker ball, visible for the tracking system, by coupling the two sensor systems tightly together. The algorithm is evaluated in simulation and tested with an experimental hardware platform. The combined sensor system is robust with respect to short time marker occlusions and effectively compensates for latencies in the pose measurements.

Telesurgery systems integrate multimodal communication and robotic technologies to enable surgical procedures to be performed from remote locations. They allow human surgeons to intuitively control laparoscopic instruments and to navigate within the human body. In this paper, we present selected topics on multimodal interaction in the context of telesurgery applications. These are results from the collaborative research project SFB 453 on ldquoHigh-Fidelity Telepresence and Teleactionrdquo which is funded by the German Research Foundation in the larger Munich area. The focus in this paper is on multimodal information processing and communication including simulation of surgical targets in the human body. Furthermore, we present an overview of our advanced multimodal telesurgery demonstrators that provide a comprehensive platform for our collaborative telepresence research.