Teleoperated robotic systems have become instrumental in advancing remote healthcare services, especially in tasks that require precision and expert oversight. The advent of cutting-edge telecommunication infrastructures, such as 5G, has amplified interest in these systems, although their full potential remains untapped. This study delves into the effectiveness of teleoperated robotic systems for medicine dispensing, comparing the performance of Wi-Fi and 5G networks in a transnational setup between two cities - Prague and Munich. We focus on the robot's ability to accurately dispense a predefined volume of a syrup-like substance, simulating a delicate healthcare operation, under the guidance of a distant operator. Our research examines the system's holistic performance in real-world implementation across diverse scenarios, encompassing varying network states and feedback methods. Two primary feedback scenarios are considered: one incorporating real-time video streaming and another offering explicit quantitative data on the dispensed volume. Using a blend of quantitative and qualitative methods, we aim to determine the influence of network type and feedback on task efficacy and user satisfaction. This study provides insights into the potential and hurdles of deploying teleoperated robotic systems in crucial healthcare contexts, guiding future advancements in this domain, especially in scenarios, where precision and dependability are crucial.

We propose a novel scene-robot interaction graph (SRI-Graph) that exploits the known position of a mobile manipulator for robust and accurate scene understanding. Compared to the state-of-the-art scene graph approaches, the proposed SRI-Graph captures not only the relationships between the objects, but also the relationships between the robot manipulator and objects with which it interacts. To improve the detection accuracy of spatial relationships, we leverage the 3D position of the mobile manipulator in addition to RGB images. The manipulator's ego information is crucial for a successful scene understanding when the relationships are visually uncertain. The proposed model is validated for a real-world 3D robot-assisted feeding task. We release a new dataset named 3DRF-Pos for training and validation. We also develop a tool, named LabelImg-Rel, as an extension of the open-sourced image annotation tool LabelImg for a convenient annotation in robot-environment interaction scenarios*. Our experimental results using the Movo platform show that SRI-Graph outperforms the state-of-the-art approach and improves detection accuracy by up to 9.83%.

We propose a novel interactive semantic shared control framework that exploits an active high-level communication loop between the human operator and the robot for time-efficient teleoperation. In shared control approaches, accurate prediction of the operator’s intention is crucial to enable the robot to provide meaningful assistance. Incorrect intention prediction (e.g., target objects to be interacted with) increases the task duration due to conflicts between human behaviors and robot guidance. Unlike existing methods, our approach not only passively observes and predicts the human operator’s input in the haptic control loop, but also actively communicates with the human operator in an additional semantic loop in the form of a speech user interface to optimize the effectiveness of assistance. We evaluate our ISSC framework for a pegin-hole teleoperation task. The experimental results show that the proposed framework significantly outperforms teleoperation without assistance and conventional shared control paradigms regarding task execution efficiency and user control quality, and reduces task completion time by up to 26.68% and 39.00%, respectively.

Pouring liquids accurately into containers is one of the most challenging tasks for robots as they are unaware of the complex fluid dynamics and the behavior of liquids when pouring. Therefore, it is not possible to formulate a generic pouring policy for real-time applications. In this paper, we propose PourNet, as a generalized solution to pouring different liquids into containers. PourNet is a hybrid planner that uses deep reinforcement learning, for end-effector planning, and Nonlinear Model Predictive Control, for joint planning. In this work, we introduce a novel simulation environment using Unity3D and NVIDIA-Flex to train our agents. By effective choice of the state space, action space and the reward functions, we allow for a direct sim-to-real transfer of the learned skills without additional training. In the simulation, PourNet outperforms state-of-the-art by an average of 4.9g deviation for water-like, and 9.2g deviation for honey-like liquids. In the real-world scenario using Kinova Movo Platform, PourNet achieves an average pouring deviation of 2.3g for dish soap when using a novel pouring container. The average pouring deviation measured for water was 5.5g. All comprehensive experiments and the simulation environment is available at: http://cxdcxd.github.io/RRS/.

Advanced wireless communication networks provide lower latency and a higher transmission rate. Although this is an enabler for many new teleoperation applications, the risk of network instability or packet drop is still unavoidable. Real-time manipulator teleoperation requires data transmission with no discontinuity. Shared autonomy (SA) is a standard method to mitigate this issue. In this way, if the data from the remote side is unavailable, the controller can continue based on the previously observed models. However, due to the spatial gap between human and robot trajectories, indisputable fluctuations occur, which cause issues in teleoperation applications. This motivates us to propose a new skill refinement strategy to modify the previously trained skill and mitigate the sudden unwanted motions within the control takeover phase. To this end, our approach comprises applying the Hidden Semi-Markov Model (HSMM) and Linear Quadratic Tracker (LQT) in combination to learn and predict the user's intentions and then exploiting Coherent Point Drift (CPD) to refine the executable trajectory. We test our method both in simulation and in the real world for 2D English letter drawing and 3D robot-assisted feeding scenarios. Our experimental results using the Kinova® Movo platform show that the proposed refinement approach generates a stable trajectory and mitigates the control switching inconsistency. All comprehensive experiments and source code is available at: http://cxdcxd.github.io/SkillCPD.

Motion control and planning for the manipulator are critical components in manipulator teleoperation. Online (real-time) motion control is challenging for active obstacle avoidance and often results in fluctuating and unsafe motion. Offline motion planning, on the other hand, generates precise and secure trajectories for complex manipulation. In this paper, a real-time nonlinear model predictive control based motion planner (NMPC-MP) is designed for teleoperated manipulation. In contrast to traditional NMPC-based approaches, our model considers a complex environment with dynamic obstacles. Our multi-threaded NMPC-MP allows for real-time planning, including dynamic objects. We evaluate our approach both in a simulated environment and with real-world experiments using the Kinova®Movo platform. The comparison to state-of-the-art approaches (e.g., RRT-Connect, CHOMP, and STOMP) shows a significant improvement in real-time motion planning using NMPC-MP. In real-world tests, the proposed planner was applied on a human-shaped dual manipulator setup. Our results show that the NMPC-MP runs in real-time and generates smooth and reliable trajectories. The experiments validate that the planner is able to precisely track active goals from the teleoperator while avoiding self-collision and obstacles.