Telesurgery systems integrate multimodal communication and robotic technologies to enable surgical procedures to be performed from remote locations. They allow human surgeons to intuitively control laparoscopic instruments and to navigate within the human body. In this paper, we present selected topics on multimodal interaction in the context of telesurgery applications. These are results from the collaborative research project SFB 453 on ldquoHigh-Fidelity Telepresence and Teleactionrdquo which is funded by the German Research Foundation in the larger Munich area. The focus in this paper is on multimodal information processing and communication including simulation of surgical targets in the human body. Furthermore, we present an overview of our advanced multimodal telesurgery demonstrators that provide a comprehensive platform for our collaborative telepresence research.

The term scaffolding, with respect to human education, was first coined in the 1970ies, although the basic concept originates back to the 1930ies. The main idea is to formalize the superior knowledge of a teacher in a certain way to generate support for a trainee. In practice, this concept can be implemented as concrete as a cloze, which assists pupils in learning a foreign language, or it might be as abstract as a social environment, which facilitates learning of specific tasks. This paper introduces a novel approach towards robotic learning by means of such a scaffolding framework. In this case, the scaffolding is constituted by abstract patterns, which facilitate the structuring and segmentation of information during "Learning by Demonstration". The methodology was applied to a real-world scenario of robot-assisted surgery.

A popular method for an easy and also flexible programming of robots is learning by demonstration. An intelligent controller learns a task from several examples carried out by an experienced user. Afterwards, the task can be adapted to new, formerly unknown environments. One particular challenge arising with this technique is generalization of demonstrations in order to get a generic description of the task. In this paper a new methodology for solving this problem is proposed. The main part of the algorithm exploits principles known from fluid dynamics.

Tying suture knots is a time-consuming task performed frequently during minimally invasive surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between identical inputs that require different actions at different points along a trajectory, it would be impossible to teach the system using traditional feedforward neural nets or support vector machines. Instead we exploit more powerful, recurrent neural networks (RNNs) with adaptive internal states. Results obtained using LSTM RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control

In this paper we present an application of robotic systems in minimally invasive surgery. We have assembled a bi-manual telemanipulator capable of delicate operations. Each robotic arm has eight degrees of freedom which enables free manipulation via trocar kinematics. In addition minimally invasive instruments were equipped with strain gauge force sensors that can measure forces along three translational directions of the instrument's shaft. Forces are displayed to the user by means of two haptic devices which are also used for input. An endoscopic stereo-camera provides pictures from the region of operation to the user. In order to emulate a stereoscopic impression, pictures can be displayed by means of either a head mounted display, a CRT-screen equipped with shutter technology or a 3D video projection. With this system we have assessed the quality of force feedback and we have performed typical operations known from cardiac surgery. We also have shown that it is possible to perform certain recurrent tasks (like knot tying) automatically. Within this context we have implemented two application examples. In the first example a surgical knot is recorded and afterwards replayed with increased speed at the same position. The second example is the instantiation of a predefined knot at an arbitrary position.

Coronary angiography has become an important tool in modern medicine by providing an additional modality for heart disease diagnostics. Even though the recorded images (or image sequences) are digitally available, their analysis is very elementary in nature in most cases: the surgeon studies the angiograms before the operation, and operates without having them available in the operation room. It is up to the surgeon's memory to accurately remember the relevant details like locations of stenosis or occlusion. To improve both the analysis and the on-site availability of angiograms,but also to drastically increase the usefulness of the angiogram as a diagnostical modality, we present a novel approach which combines several advanced techniques into a single tool allowing real-time augmentation of in situ beating heart image sequences with the corresponding angiogram. The software toolset we developed so far contains the removal of specularites from the glossy heart surface and the reconstruction of the original image colors, color based coronary artery segmentation in beating heart image sequences, angiogram segmentation and symbolic representation, OpenGL based augmentation and visualization, mass-spring-NURBS-based soft tissue simulation. Future work will focus on developing robust matching techniques and on integrating the software tools into an operation room suitable hardware setup.

Minimally invasive robotic surgery systems has entered daily practice in dedicated clinical centers. Especially heart surgery profits from this new technique, due to a higher accuracy compared to conventional endoscopic interventions. Nevertheless some drawbacks have restricted a broader acceptance of these devices. The most urgent issues are lack of haptic feedback and prolonged operation time. Our research project tackles both topics, on the one hand by adding high fidelity force-feedback, on the other hand by automating recurrent manipulation tasks. These features have been integrated into the Endo[PA]R (endoscopic partially-autonomous robot) system, an open evaluation platform for robotic surgery. The setup comprises two low-payload robots equipped with sensorized surgical instruments and a third robot carrying a stereo camera system. Trocar kinematics, enabling surgical manipulation through small incisions, has been implemented for all robotic arms. In order to ensure operation close to reality, a thorax and heart phantom for surgical training was used in the experiments. Stereo vision is provided via a head-mounted display and force-reflective input devices are employed for user interaction. The system was evaluated by surgeons and it was capable of performing autonomous knot-tying.