Since its official introduction in 2012, the Robot Web Tools project has grown tremendously as an open-source community, enabling new levels of interoperability and portability across heterogeneous robot systems, devices, and front-end user interfaces. At the heart of Robot Web Tools is the rosbridge protocol as a general means for messaging ROS topics in a client-server paradigm suitable for wide area networks, and human-robot interaction at a global scale through modern web browsers. Building from rosbridge, this paper describes our efforts with Robot Web Tools to advance: 1) human-robot interaction through usable client and visualization libraries for more efficient development of front-end human-robot interfaces, and 2) cloud robotics through more efficient methods of transporting high-bandwidth topics (e.g., kinematic transforms, image streams, and point clouds). We further discuss the significant impact of Robot Web Tools through a diverse set of use cases that showcase the importance of a generic messaging protocol and front-end development systems for human-robot interaction.

Given the proliferation of consumer media recording devices, events often give rise to a large number of recordings. These recordings are taken from different spatial positions and do not have reliable timestamp information. In this paper, we present two robust graph-based approaches for synchronizing multiple audio signals. The graphs are constructed atop the over-determined system resulting from pairwise signal comparison using cross-correlation of audio features. The first approach uses a Minimum Spanning Tree (MST) technique, while the second uses Belief Propagation (BP) to solve the system. Both approaches can provide excellent solutions and robustness to pairwise outliers, however the MST approach is much less complex than BP. In addition, an experimental comparison of audio features-based synchronization shows that spectral flatness outperforms the zero-crossing rate and signal energy.

We present a novel lossy compression approach for point cloud streams which exploits spatial and temporal redundancy within the point data. Our proposed compression framework can handle general point cloud streams of arbitrary and varying size, point order and point density. Furthermore, it allows for controlling coding complexity and coding precision. To compress the point clouds, we perform a spatial decomposition based on octree data structures. Additionally, we present a technique for comparing the octree data structures of consecutive point clouds. By encoding their structural differences, we can successively extend the point clouds at the decoder. In this way, we are able to detect and remove temporal redundancy from the point cloud data stream. Our experimental results show a strong compression performance of a ratio of 14 at 1 mm coordinate precision and up to 40 at a coordinate precision of 9 mm.

In this video we present a teleoperation system which is capable of solving complex tasks in human-sized wide area environments. The system consists of two mobile teleoperators controlled by two operators, and offers haptic, visual, and auditory feedback. The task examined here, consists of repairing a robot by removing a computer and replacing a defective hard-drive. To cope with the complexity of such a task, we go beyond classical teleoperation by integrating several advanced software algorithms into the system.

Audiovisual communications is at the core of multimedia systems that allow users to interact across distances. It is common understanding that both audio and video are required for high-quality interaction. While audiovisual information provides a user with a satisfactory impression of being present in a remote environment, physical interaction and manipulation is not supported. True immersion into a distant environment and efficient distributed collaboration require the ability to physically interact with remote objects and to literally get in touch with other people. Touching and manipulating objects remotely becomes possible if we augment traditional audiovisual communications by the haptic modality. Haptic communications is a relatively young field of research that has the potential to substantially improve human-human and human-machine interaction. In this paper, we discuss the state-of-the-art in haptic communications both from psychophysical and technical points of view. From a human perception point of view, we mainly focus on the multimodal integration of video and haptics and the improved performance that can be achieved when combining them. We also discuss how the human adapts to discrepancies and synchronization errors between different modalities, a research area which is typically referred to as perceptual learning. From a technical perspective, we address perceptual coding of haptic information and the transmission of haptic data streams over resource-constrained and potentially lossy networks in the presence of unpredictable and time-varying communication delays. In this context, we also discuss the need for objective quality metrics for haptic communication. Throughout the paper, we stress the fact that haptic communications is not meant as a replacement of traditional audiovisual communications but rather as an additional dimension for telepresence that will allow us to advance in our quest for truly immersive communication.

In this paper, we propose a novel framework for the efficient recording and playback of visual-haptic telepresence sessions. During recording, all involved modalities are synchronously encoded and stored. During replay, interaction with the tele-environment is no longer possible and the displayed force feedback does not relate to the user's interaction anymore. Furthermore, position guidance by applying force-feedback prevents the presentation of the recorded haptic impressions. In our experiments, we have discovered that the human operator can easily associate the previously recorded movements and actions by following the endeffector position of the teleoperator in the displayed visual recording and relate the presented force feedback to its observation. We describe and investigate this observation which allows the user to immerse into the previously recorded telemanipulation session. Conducted psychophysical experiments investigate the performance of our framework in conjunction with perceptual offline coding for haptic signals. Our user studies reveal that our approach can provide high-fidelity visual-haptic experiences for learning, entertainment and/or performance analysis purposes.

The performance of haptic interaction across communication networks critically depends on the successful reconstruction of the bidirectionally transmitted haptic signals, and hence on the quality of the communication channel. We propose a novel error-resilient data reduction scheme for haptic communication which exploits known limits of human haptic perception. Particularly, we show that missing haptic information due to packet loss may strongly impair the user's experience during haptic interaction. We present and compare methods that eliminate the disturbing artifacts resulting out of packet loss. Our approach keeps the estimated impact of packet losses below human perception thresholds. A tree of possible cases (packets received or not received) and their respective occurrence probabilities is maintained at the sender side, and the system predicts unacceptable error cases to decide whether extra packets should be sent. We introduce different criteria that can be employed to trigger additional packets. In our experiments, we evaluate both the objective data reduction performance and the subjective system transparency by performing extensive tests using packet loss probability and round trip time as parameters. The proposed scheme shows excellent performances in terms of data reduction while sustaining good subjective ratings for a wide range of packet loss values and round trip times.

Telesurgery systems integrate multimodal communication and robotic technologies to enable surgical procedures to be performed from remote locations. They allow human surgeons to intuitively control laparoscopic instruments and to navigate within the human body. In this paper, we present selected topics on multimodal interaction in the context of telesurgery applications. These are results from the collaborative research project SFB 453 on ldquoHigh-Fidelity Telepresence and Teleactionrdquo which is funded by the German Research Foundation in the larger Munich area. The focus in this paper is on multimodal information processing and communication including simulation of surgical targets in the human body. Furthermore, we present an overview of our advanced multimodal telesurgery demonstrators that provide a comprehensive platform for our collaborative telepresence research.

We present a theoretical analysis of a perceptual coding approach, the so called Weber quantizer. Extensive studies performed by experimental psychologists and physiologists have unveiled one major conclusion: human perception often follows Weber's law. Ernst Weber was an experimental physiologist who in 1834 first discovered the following implication DeltaI = kl, where DeltaI is the so called difference threshold or the just noticeable difference (JND). It describes the smallest amount of change of an (arbitrary) stimulus I which can be detected just as often as it cannot be detected and defines the Weber bound at [(1 - k)I, (1 + k)I].

In this work, a novel perceptual coding approach for offline compression of haptic media is presented. Our scheme exploits the properties of human haptic perception and hides coding artifacts introduced by lossy compression below the human perception thresholds. We combine the concept of Just Noticeable Differences with predictive coding in order to achieve high coding efficiency without impairing user perception. In our experiments, we apply the proposed lossy compression scheme to haptic data that has been recorded during a telemanipulation session in a virtual environment. Our user studies reveal that our approach leads to strong data reduction up to 100:1 while preserving a high-quality haptic experience during playback of the compressed haptic data streams.

Subband modulation spectra (SB-MS) have proven to be useful for many applications in audio signal analysis and audio signal processing. This paper presents a novel blind watermarking scheme for embedding and detecting a watermark in the subband modulation spectral domain representation of an audio signal. Some results of audio quality, data rate and robustness are presented. The scheme is shown to be robust to a number of attacks while providing excellent subjective audio quality.

The performance and immersiveness of telepresence and teleaction systems critically depend on the quality of the communication between the operator and the teleoperator. High packet rate for low delay data exchange in internet-based tele-operation is one of the major challenges in this context. A psychophysical^ motivated data reduction scheme, earlier presented by the authors, solves this problem by the use of signal amplitude dependent deadband quantization. This scheme successfully limits the amount of transmitted packets, but as soon as the size of the deadband exceeds a certain limit, noticeable and disturbing artifacts are introduced into the haptic signals. In this paper, we present a novel signal reconstruction technique, based on a signal adaptive synthesis filter. It removes discontinuities in the displayed signals and enables the use of strong deadband based data reduction without perceptible disturbance in the telepresence system.