In this paper, we present a targeted training approach for remote teleoperation aimed at achieving consistent proficiency levels across users with varying capabilities. Our approach begins by assessing users’ abilities to perform robot motion control, workspace adaptation, and gripper control. It then provides tailored training based on identified skill gaps to enhance the learning effectiveness and user experience. To demonstrate our approach, we conducted a user study, with one group undergoing conventional, free-form training and the other engaging in targeted training in accordance with their skill gaps; after the training phase, participants teleoperated a robotic arm in a simulated medication preparation task for performance evaluation. Our results show that the targeted training approach effectively reduces performance variability and mitigates the influence of spatial ability on both training and task completion time. We discuss the implications of our results for practical teleoperation training and future research.

Teleoperation enables controlling complex robot systems remotely, providing the ability to impart human expertise from a distance. However, these interfaces can be complicated to use as it is difficult to contextualize information about robot motion in the workspace from the limited camera feedback. Thus, it is required to study the best manner in which assistance can be provided to the operator that reduces interface complexity and effort required for teleoperation. Some techniques that provide assistance to the operator while freeform teleoperating include: (1) perception augmentation, like augmented reality visual cues and additional camera angles, increasing the information available to the operator; (2) action augmentation, like assistive autonomy and control augmentation, optimized to reduce the effort required by the operator while teleoperating. In this article, we investigate: (1) which aspects of dexterous telemanipulation require assistance; (2) the impact of perception and action augmentation in improving teleoperation performance; and (3) what factors impact the usage of assistance and how to tailor these interfaces based on the operators’ needs and characteristics. The findings from this user study and resulting post-study surveys will help identify task-based and user-preferred perception and augmentation features for teleoperation assistance.

When humans control or supervise remote robot manipulation, augmented reality (AR) visual cues overlaid on the remote camera video stream can effectively enhance human's remote perception of task and robot states, and comprehension of the robot autonomy's capability and intent. In this work, we conducted a user study (N=18) to investigate: (RQ1) what AR cues humans prefer when controlling the robot with various levels of autonomy, and (RQ2) whether this preference can be influenced by the way humans learn to use the interface. We provided AR visual cues of various types (e.g., motion guidance, obstacle indicator, target hint, autonomy activation and intent) to assist humans to pick and place an object around an obstacle on a counter workspace. We found that: 1) Participants prefer different types of AR cues based on the level of robot autonomy; 2) The AR cues the participants prefer to use after hands-on robot operation converged to the recommendation of experienced users, and may largely differ from their initial selection based on video instruction.

Teleoperation enables complex robot platforms to perform tasks beyond the scope of the current state-of-the-art robot autonomy by imparting human intelligence and critical thinking to these operations. For seamless control of robot platforms, it is essential to facilitate optimal situational awareness of the workspace for the operator through active telepresence cameras. However, the control of these active telepresence cameras adds an additional degree of complexity to the task of teleoperation. In this paper we present our results from the user study that investigates: (1) how the teleoperator learns or adapts to performing the tasks via active cameras modeled after camera placements on the TRINA humanoid robot; (2) the perception-action coupling operators implement to control active telepresence cameras, and (3) the camera preferences for performing the tasks. These findings from the human motion analysis and post-study survey will help us determine desired design features for robot teleoperation interfaces and assistive autonomy.

Robot teleoperation via human motion tracking has been proven to be easy to learn, intuitive to operate, and facilitate faster task execution than existing baselines. However, precise control while performing the dexterous telemanipulation tasks is still a challenge. In this paper, we implement sensory augmentation in terms of haptic and augmented reality visual cues to represent four types of information critical to the precision and performance of a telemanipulation task, namely: (1) target location; (2) constraint alert; (3) grasping affordance; and (4) grasp confirmation. We further conduct two user studies to investigate the effectiveness and preferred modality of the sensory feedback against no sensory support, and how the preference will be influenced by the different types of simulated real-world additional workload. We asked 8 participants to perform a general manipulation task using a KINOVA robotic arm. Our results indicate that: (1) the haptic and AR visual cues can significantly reduce the task completion time, occurrences of errors, the total length traversed by the robot end-effector, the operational effort while increasing the interface usability; (2) the haptic feedback trended in the direction of presenting the information that needs a prompt response, while the AR visual cues are suitable to monitor the system status; (3) the participants chose their preferred feedback with the purpose of reducing the cognitive workload despite increased extra effort.

Motion tracking interfaces are intuitive for free-form teleoperation tasks. However, efficient manipulation control can be difficult with such interfaces because of issues like the interference of unintended motions and the limited precision of human motion control. The limitation in control efficiency reduces the operator's performance and increases their workload and frustration during robot teleoperation. To improve the efficiency, we proposed separating controlled degrees of freedom (DoFs) and adjusting the motion scaling ratio of a motion tracking interface. The motion tracking of handheld controllers from a Virtual Reality system was used for the interface. We separated the translation and rotational control into: 1) two controllers held in the dominant and non-dominant hands and 2) hand pose tracking and trackpad inputs of a controller. We scaled the control mapping ratio based on 1) the environmental constraints and 2) the teleoperator's control speed. We further conducted a user study to investigate the effectiveness of the proposed methods in increasing efficiency. Our results show that the separation of position and orientation control into two controllers and the environment-based scaling methods perform better than their alternatives.

Tele-nursing robots provide a safe approach for patient-caring in quarantine areas. For effective nurse–robot collaboration, ergonomic teleoperation and intuitive interfaces with low physical and cognitive workload must be developed. We propose a framework to evaluate the control interfaces to iteratively develop an intuitive, efficient, and ergonomic teleoperation interface. The framework is a hierarchical procedure that incorporates general to specific assessment and its role in design evolution. We first present pre-defined objective and subjective metrics used to evaluate three representative contemporary teleoperation interfaces. The results indicate that teleoperation via human motion mapping outperforms the gamepad and stylus interfaces. The tradeoff with using motion mapping as a teleoperation interface is the non-trivial physical fatigue. To understand the impact of heavy physical demand during motion mapping teleoperation, we propose an objective assessment of physical workload in teleoperation using electromyography. We find that physical fatigue happens in the actions that involve precise manipulation and steady posture maintenance. We further implemented teleoperation assistance in the form of shared autonomy to eliminate the fatigue-causing component in robot teleoperation via motion mapping. The experimental results show that the autonomous feature effectively reduces the physical effort while improving the efficiency and accuracy of the teleoperation interface.

Robot teleoperation is a reliable way to perform a variety of tasks with complex robotic systems. However, the remote control of active telepresence cameras on the robot for improved telepresence adds an additional degree of complexity while teleoperating and can thus affect the operator’s performance during tele-manipulation. Our previous user study investigates the general human performance and preference when using various wearable cameras. In this paper, we further investigate how humans respond to the usage of telepresence cameras in terms of motion behavior. The findings from our human motion analysis inform several desired designs for robot teleoperation interfaces and assistive autonomy.

Motion mapping is an intuitive method of teleoperation with a low learning curve. Our previous study investigates the physical fatigue caused by teleoperating a robot to perform general-purpose assistive tasks and this fatigue affects the operator’s performance. The results from that study indicate that physical fatigue happens more in the tasks which involve more precise manipulation and steady posture maintenance. In this paper, we investigate how teleoperation assistance in terms of shared autonomy can reduce the physical workload in robot teleoperation via motion mapping. Specifically, we conduct a user study to compare the muscle effort in teleoperating a mobile humanoid robot to (1) reach and grasp an individual object and (2) collect objects in a cluttered workspace with and without an autonomous grasping function that can be triggered manually by the teleoperator. We also compare the participants’ task performance, subjective user experience, and change in attitude towards the usage of teleoperation assistance in the future based on their experience using the assistance function. Our results show that: (1) teleoperation assistance like autonomous grasping can effectively reduce the physical effort, task completion time and number of errors; (2) based on their experience performing the tasks with and without assistance, the teleoperators reported that they would prefer to use automated functions for future teleoperation interfaces.

Robot teleoperation via motion mapping has been demonstrated to be an efficient and intuitive approach for controlling and teaching the whole-body motion coordination of humanoid robots. However, the physical fatigue in the usage of such robot teleoperation interfaces may prevent this approach to be widely used in large scale by diverse workforce populations. As a result, this paper conducts a user study to investigate the physical fatigue of teleoperators in the whole-body motion mapping teleoperation of a mobile humanoid assistive robot. Through a Vicon motion capture system, participants teleoperated the robot to perform general purpose assistive tasks that involve reaching-to-grasp, bimanual manipulation, loco-manipulation and human-robot interaction. We assess the physical fatigue based on surface electromyography (sEMG) measurement, and compare it between different tasks and muscles. Our analysis results indicate that: (1) Fatigue happens more in the tasks that involve more precise manipulation and steady posture maintenance; (2) Deltoids, Biceps and Trapezius are used more for such tasks and thus have more fatigue than others. These findings imply that automating the fatigue-causing task components may reduce the physical fatigue in motion mapping teleoperation.