This paper presents a comprehensive characterization of kinaesthetic data traffic and associated performance trade-offs in multimodal bilateral haptic teleoperation over 5G networks. Utilizing two widely adopted haptic devices interacting with virtual environments, precise telemetry on motion and force was obtained across various interaction types and virtual object properties. The study investigates the effects of network resource allocation within a 5G standalone private network by employing three predefined network slicing profiles prioritizing haptic, audio, or video traffic. Performance evaluations were conducted using Quality of Service (QoS) metrics, from which potential implications for Quality of Experience (QoE) were subsequently inferred. The results show the interplay between traffic patterns and resource allocation, identifying configurations that ensure high-fidelity haptic feedback while maintaining acceptable audiovisual quality. These insights lay the groundwork for future research on adaptive network management strategies for haptic-enabled teleoperation systems.

We propose a novel scene-robot interaction graph (SRI-Graph) that exploits the known position of a mobile manipulator for robust and accurate scene understanding. Compared to the state-of-the-art scene graph approaches, the proposed SRI-Graph captures not only the relationships between the objects, but also the relationships between the robot manipulator and objects with which it interacts. To improve the detection accuracy of spatial relationships, we leverage the 3D position of the mobile manipulator in addition to RGB images. The manipulator's ego information is crucial for a successful scene understanding when the relationships are visually uncertain. The proposed model is validated for a real-world 3D robot-assisted feeding task. We release a new dataset named 3DRF-Pos for training and validation. We also develop a tool, named LabelImg-Rel, as an extension of the open-sourced image annotation tool LabelImg for a convenient annotation in robot-environment interaction scenarios*. Our experimental results using the Movo platform show that SRI-Graph outperforms the state-of-the-art approach and improves detection accuracy by up to 9.83%.

This paper proposes a novel Generative Adversarial Network (GAN)-based strategy to augment subjective haptic Quality of Experience (QoE) datasets for bilateral teleoperation with haptic feedback without conducting time-consuming subjective experiments. In our previous work, we proposed a multi-assessment fusion approach to predict subjective haptic quality using a collection of objective metrics. This method requires a sufficiently large haptic dataset with QoE labels. The proposed generative approach automatically expands the existing haptic quality dataset by combining a modified conditional GAN (CGAN) and Style GAN (StyleGAN) architecture. The most important feature of our method is that it learns from the labeled training data and focuses on synthesizing signals with artifacts according to new input labels containing the QoE score, time delay, control method, and data reduction information. Extensive experiments are conducted to validate the suitability of the expanded dataset. The results show that our approach is able to generate new data, which match the label and signal distribution of the original data with categorical rank and linear correlation of over 0.85.

This paper proposes a Quality-of-Task-Perception (QoTP) based performance optimization approach for bilateral haptic teleoperation. For time-delayed teleoperation, stabilizing control schemes are combined with communication and data reduction algorithms to ensure stability, transparency, and Quality of Experience (QoE). An adaptive control scheme switching strategy to improve the QoE of teleoperation considering network quality of service (QoS) and quality of control (QoC) is proposed in our previous work. In this paper, we introduce a novel concept named quality of task perception (QoTP) to optimize teleoperation from another dimension in addition to QoS and QoC. QoTP represents the pre-cognition of the task and the accuracy of the environment restoration. The proposed optimization approach is applied to a haptic teleoperation system with switchable control schemes (prediction-based or passivity-based). An environment restoration model is set on the leader side using the least squares method (LSM) to fit different environment models and provide force feedback without the influence of round-trip delay. We also evaluate the system performance with different delays, control schemes, and model complexities both objectively and subjectively. Our experiments validate the proposed approach and show that the QoE performance increases when selecting the more accurate environment restoration model in the QoTP dimension considering the system’s computing power.

We propose a novel interactive semantic shared control framework that exploits an active high-level communication loop between the human operator and the robot for time-efficient teleoperation. In shared control approaches, accurate prediction of the operator’s intention is crucial to enable the robot to provide meaningful assistance. Incorrect intention prediction (e.g., target objects to be interacted with) increases the task duration due to conflicts between human behaviors and robot guidance. Unlike existing methods, our approach not only passively observes and predicts the human operator’s input in the haptic control loop, but also actively communicates with the human operator in an additional semantic loop in the form of a speech user interface to optimize the effectiveness of assistance. We evaluate our ISSC framework for a pegin-hole teleoperation task. The experimental results show that the proposed framework significantly outperforms teleoperation without assistance and conventional shared control paradigms regarding task execution efficiency and user control quality, and reduces task completion time by up to 26.68% and 39.00%, respectively.

Advanced wireless communication networks provide lower latency and a higher transmission rate. Although this is an enabler for many new teleoperation applications, the risk of network instability or packet drop is still unavoidable. Real-time manipulator teleoperation requires data transmission with no discontinuity. Shared autonomy (SA) is a standard method to mitigate this issue. In this way, if the data from the remote side is unavailable, the controller can continue based on the previously observed models. However, due to the spatial gap between human and robot trajectories, indisputable fluctuations occur, which cause issues in teleoperation applications. This motivates us to propose a new skill refinement strategy to modify the previously trained skill and mitigate the sudden unwanted motions within the control takeover phase. To this end, our approach comprises applying the Hidden Semi-Markov Model (HSMM) and Linear Quadratic Tracker (LQT) in combination to learn and predict the user's intentions and then exploiting Coherent Point Drift (CPD) to refine the executable trajectory. We test our method both in simulation and in the real world for 2D English letter drawing and 3D robot-assisted feeding scenarios. Our experimental results using the Kinova® Movo platform show that the proposed refinement approach generates a stable trajectory and mitigates the control switching inconsistency. All comprehensive experiments and source code is available at: http://cxdcxd.github.io/SkillCPD.

This work proposes a novel haptic data reduction scheme for time-delayed teleoperation by coding information as blocks. State-of-the-art (SOTA) haptic data reduction approaches are mainly sampled-based schemes. They encode haptic signals sample by sample in order to minimize the introduced coding delay. In contrast, our proposed block-based coding approach transmits a sample block as a single unit (haptic packet). Although it introduces additional algorithmic delays that are proportional to the block length, block coding has benefits since the packet rate is easy to control, the coding approach can be lossless, and the intra-block information can be employed to improve the force feedback quality. We further develop an energy adjustment approach that uses the information in a block to mitigate force oscillations caused by the Time Domain Passivity Approach. Simulation experiments and subjective tests demonstrate that our method reduces network load and significantly increases force feedback quality compared with the SOTA sample-based coding schemes, particularly for mid- to high-latency networks and low packet rates.

In this paper, we propose a robust depth estimation strategy that uses RGB images and mmWave radar data to deal with limited visibility in foggy environments. While the state-of-the-art RGB or LiDAR-based depth estimation works well in scenarios with good visibility, their performance dramatically degrades in the presence of fog. In contrast, mmWave radar sensors are not affected by fog and hence are a promising complement. To leverage this property of mmWave radar, we combine RGB image-based depth estimation with radar information. The proposed combination is an extension of the Sparse-to-Dense (S2D) model. Moreover, a weight-based sensor fusion strategy is presented to improve system performance. Our experiments show that a fog density of meteorological optical range (MOR) less than 50m leads to strongly degraded performance for RGB image-based and LiDAR-based depth estimation. For a MOR of 30m in our dataset, the experiments show an improvement of 26% in mean square error for our proposed approach compared to the combination of RGB images and LiDAR data.

This paper presents a novel quality assessment approach for the prediction of the subjective haptic experience in time-delayed teleoperation. With the rapid development of haptic technology in remote robot control and virtual reality, new control schemes and hardware systems are developed to provide high quality human-in-the-loop teleoperation service. Our subjective experiments indicate that the existing objective quality assessment metrics do not sufficiently correlate with the subjective haptic experience of the users. This gap requires expensive and time-consuming subjective experiments to be conducted. To avoid time-consuming experiments and provide a fast and accurate subjective experience prediction, we make an attempt to analyze and explain the mismatch between the subjective and objective haptic signal quality metrics. To this end, extensive subjective experiments and case studies have been conducted for teleoperation with time delay and haptic data reduction. Based on our experimental results, we propose a quality assessment approach that predicts the subjective quality of experience using multiple objective metrics. For the one-dimensional spring model, the Spearman’s rank-order, Kendall’s rank-order and Pearson’s Linearity correlation coefficient (SROCC, KLOCC and PLCC) between the predictions of our model and the results of subjective experiment show remarkable improvement on the correlation between subjective and objective quality assessment.

Teleoperation systems with haptic feedback allow a human user to remotely interact with a dangerous or inac-cessible environment, perform various tasks, and perceive the haptic feedback. To ensure system stability while maintaining the best possible quality of experience (QoE), different teleoperation control schemes and haptic communication strategies need to be selected to adapt to varying network conditions and teleoperation tasks. In this paper, we propose a QoE-driven control scheme switching approach, which adaptively selects the control scheme that provides the best possible QoE for varying communication delay. A transition period is designed to moderate the artifacts during the switching phase. Haptic data reduction approaches are developed for the switching strategy to match the characteristics of each control scheme. Our experiments verify the feasibility of the proposed scheme. Subjective tests confirm that the proposed adaptive switching scheme is able to achieve a superior user QoE in contrast to a fixed control scheme in the presence of varying communication delay up to 200 ms.

Transfer of skills and teaching tasks to robots face new challenges when the demonstrations are provided remotely via tele-operation. Not only having a remote operator, but also the communication between the tele-operator and the operator affects the quality of demonstrations. Artifacts introduced by lossy haptic data compression and communication delay deteriorate the system transparency; however, the impact of these on the quality of learning has not been studied yet. In this paper, we construct the bridge between the learning quality and the reduced transparency caused by lossy haptic data compression during teleoperation with haptic feedback. The considered haptic data compression scheme is the previously proposed perceptual dead band-based kinesthetic data reduction approach. The learning quality is assessed both with the mean squared error (MSE) metric on the trajectory level and with the rate of success defined on the task requirement. Our experiments show that the learning quality is reduced significantly for a dead band parameter larger than 20% and 30% for a cube following and peg-in-hole tasks, respectively.

This article proposes an application-layer multiplexing scheme for teleoperation systems with multimodal feedback (video, audio, and haptics). The available transmission resources are carefully allocated to avoid delay-jitter for the haptic signal potentially caused by the size and arrival time of the video and audio data. The multiplexing scheme gives high priority to the haptic signal and applies a preemptive-resume scheduling strategy to stream the audio and video data. The proposed approach estimates the available transmission rate in real time and adapts the video bitrate, data throughput, and force buffer size accordingly. Furthermore, the proposed scheme detects sudden transmission rate drops and applies congestion control to avoid abrupt delay increases and converge promptly to the altered transmission rate. The performance of the proposed scheme is measured objectively in terms of end-to-end signal latencies, packet rates, and peak signal-to-noise ratio (PSNR) for visual quality. Moreover, peak-delay and convergence time measurements are carried out to investigate the performance of the congestion control mode of the system.

In this paper we describe a client-server architecture for haptic interaction with simulated deformable objects. The computationally expensive object deformation is computed on the server at a low temporal update rate and transmitted to the clients. There, an intermediate representation of the deformable object is used to locally render haptic force feedback displayed to the user at the required rate of 1 kHz. Based on a one-dimensional deformable object, we analyze the transparency of this multi-rate architecture for a single client interaction. The delay introduced by the deformation simulation and the client-server communication leads to increased rendered forces at the clients compared to a reference scenario without delay. We propose a method that adaptively adjusts the stiffness used in the local force rendering at the client to compensate for this. The evaluation shows that the proposed method successfully compensates the effect of delay in the tested delay range of up to 100 ms.

We propose a perceptual haptic data reduction approach for teleoperation systems which use the time domain passivity approach (TDPA) as their control architecture for dealing with time-varying communication delay. Our goal is to reduce the packet rate over the communication network while preserving system stability in the presence of time-varying and unknown delays. Compared to the existing wave variable-based (WV-based) haptic data reduction approaches, our proposed scheme leads to smaller distortion in the force signals and robustly deals with time-varying delays. Experiments show that our proposed approach can reduce the average packet rate by up to 80%, without introducing significant distortion. In addition, the proposed approach outperforms the existing WV-based approaches in both packet rate reduction and subjective preference for the tested communication delays.

Shared haptic virtual environments can be realized using a client-server architecture. In this architecture, each client maintains a local copy of the virtual environment (VE). A centralized physics simulation running on a server calculates the object states based on haptic device position information received from the clients. The object states are sent back to the clients to update the local copies of the VE, which are used to render interaction forces displayed to the user through a haptic device. Communication delay leads to delayed object state updates and increased force feedback rendered at the clients. In this article, we analyze the effect of communication delay on the magnitude of the rendered forces at the clients for cooperative multi-user interactions with rigid objects. The analysis reveals guidelines on the tolerable communication delay. If this delay is exceeded, the increased force magnitude becomes haptically perceivable. We propose an adaptive force rendering scheme to compensate for this effect, which dynamically changes the stiffness used in the force rendering at the clients. Our experimental results, including a subjective user study, verify the applicability of the analysis and the proposed scheme to compensate the effect of time-varying communication delay in a multi-user SHVE.

In this paper, we study and extend the concept of model-mediated teleoperation (MMT) for teleaction systems which provide live video feedback from the remote side with a time delay. In MMT, the haptic feedback is rendered locally on the operator side using a simple object surface model in order to keep the haptic control loop stable in the presence of communication delays. Because the live video from the remote side is received with delay, this results in a visual-haptic asynchrony for the displayed interaction events. In addition, sudden model parameter updates can lead to “model-jump” effects for the displayed haptic feedback. Both effects degrade the user experience and system performance. To address these issues, we propose an extension of MMT which we call model-displaced teleoperation (MDT) in this paper. In MDT, we adaptively shift the position of the local surface model to delay the haptic contact with the environment, thus compensating the visual-haptic asynchrony and avoiding the model-jump effect. As the haptic feedback is still rendered locally, the advantages of the MMT approach are retained and instabilities in the haptic interaction are avoided. In our experiments, we determine the optimal displacement compromise between visual-haptic asynchrony, the model-jump effect and perceived distance errors. Moreover, the subjective experience and objective task performance of the proposed MDT and the original MMT for a teleoperation setup with soft objects are evaluated. Our results show that the users prefer the MDT method compared to MMT once the communication delay between the teleoperator and the operator exceeds 50ms. In addition, the task error rate is reduced by about 50% and the subjects are better able to control their contact force for system delays larger than 50ms if the MDT method is employed.