Recently, many humanoid robots have been increasingly deployed in various facilities, including hospitals and assisted living environments, where they are often remotely controlled by human operators. Their kinematic redundancy enhances reachability and manipulability, enabling them to navigate complex, cluttered environments and perform a wide range of tasks. However, this redundancy also presents significant control challenges, particularly in coordinating the movements of the robot's macro-micro structure (torso and arms). Therefore, we propose various human-robot collaborative (HRC) methods for coordinating the torso and arm of remotely controlled mobile humanoid robots, aiming to balance autonomy and human input to enhance system efficiency and task execution. The proposed methods include human-initiated approaches, where users manually control torso movements, and robot-initiated approaches, which autonomously coordinate torso and arm based on factors such as reachability, task goal, or inferred human intent. We conducted a user study with$\mathbf{N} \boldsymbol{=} \mathbf{1 7}$participants to compare the proposed approaches in terms of task performance, manipulability, and energy efficiency, and analyzed which methods were preferred by participants.

Global redundancy resolution (GRR) roadmap is a novel concept in robotics that facilitates the mapping from task space paths to configuration space paths in a legible, predictable, and repeatable way. Such roadmaps could find widespread utility in applications such as safe teleoperation, consistent path planning, and motion primitives generation. However, previous methods to compute GRR roadmaps often necessitate a lengthy computation time and produce non-smooth paths, limiting their practical efficacy. To address this challenge, we introduce a novel method EXPANSION-GRR that leverages efficient configuration space projections and enables a rapid generation of smooth roadmaps that satisfy the task constraints. Additionally, we propose a simple multi-seed strategy that further enhances the final quality. We conducted experiments in simulation with a 5-link planar manipulator and a Kinova arm. We were able to generate the GRR roadmaps up to 2 orders of magnitude faster while achieving higher smoothness. We also demonstrate the utility of the GRR roadmaps in teleoperation tasks where our method outperformed prior methods and reactive IK solvers in terms of success rate and solution quality.

In this paper, we introduce an innovative robotic nursing assistance system with multilateral multimodal human-robot (MMHR) collaboration, enabling nursing robots to be assisted by remote and on-site operators. Through our augmented reality (AR) interfaces, the remote and local operators can monitor the robots’ operations, issue task and action commands, and facilitate collaborative assistance and information exchange via AR cues or verbal communication. Our preliminary user study evaluated the usability of the prototype system and validated the efficacy of our MMHR collaboration in a representative nursing assistance task scenario. The results show significant improvements in overall task efficiency for the remote operator and reveal human strategies and rationales in the spontaneous multilateral human-robot collaboration.

Teleoperation enables controlling complex robot systems remotely, providing the ability to impart human expertise from a distance. However, these interfaces can be complicated to use as it is difficult to contextualize information about robot motion in the workspace from the limited camera feedback. Thus, it is required to study the best manner in which assistance can be provided to the operator that reduces interface complexity and effort required for teleoperation. Some techniques that provide assistance to the operator while freeform teleoperating include: (1) perception augmentation, like augmented reality visual cues and additional camera angles, increasing the information available to the operator; (2) action augmentation, like assistive autonomy and control augmentation, optimized to reduce the effort required by the operator while teleoperating. In this article, we investigate: (1) which aspects of dexterous telemanipulation require assistance; (2) the impact of perception and action augmentation in improving teleoperation performance; and (3) what factors impact the usage of assistance and how to tailor these interfaces based on the operators’ needs and characteristics. The findings from this user study and resulting post-study surveys will help identify task-based and user-preferred perception and augmentation features for teleoperation assistance.

This paper presents a novel integration of a shared autonomous mobile humanoid robot for remote nursing assistance. The proposed nursing robot has a motorized versatile supporting structure to allow flexible integration of the system components, autonomously adjust its mobile manipulation workspace and improve its reachability and manipulability to operate in a cluttered environment. The robot also provides a novel integration of robot autonomy to reduce the human effort to coordinate the motorized chest and arm motion, control the precise manipulation of objects and camera viewpoint, and handle complex collision avoidance in human-guided gross manipulation. Moreover, we developed an open-source virtual testbed that integrates ROS- and Unity-based robot simulation and benchmark mobile manipulation nursing tasks and scenarios in a realistic simulation of a hospital environment. The virtual testbed supports various contemporary gaming and AR/VR interfaces to control the virtual human and robots, and provides autonomy for navigation, manipulation, and remote active perception assistance. We conducted a user study (N=9) to validate that the versatile supporting structure and shared autonomy of the physical testbed can effectively reduce the human effort to control unstructured manipulation, and improve the robot's reachability and manipulability. In addition, we conducted a pilot study (N=8) to test the usability of the virtual testbed and collect feedback from representative users.

When humans control or supervise remote robot manipulation, augmented reality (AR) visual cues overlaid on the remote camera video stream can effectively enhance human's remote perception of task and robot states, and comprehension of the robot autonomy's capability and intent. In this work, we conducted a user study (N=18) to investigate: (RQ1) what AR cues humans prefer when controlling the robot with various levels of autonomy, and (RQ2) whether this preference can be influenced by the way humans learn to use the interface. We provided AR visual cues of various types (e.g., motion guidance, obstacle indicator, target hint, autonomy activation and intent) to assist humans to pick and place an object around an obstacle on a counter workspace. We found that: 1) Participants prefer different types of AR cues based on the level of robot autonomy; 2) The AR cues the participants prefer to use after hands-on robot operation converged to the recommendation of experienced users, and may largely differ from their initial selection based on video instruction.

While motion classification architectures have improved in accuracy and robustness in recent years, computationally expensive approaches and sophisticated hardware dependencies limit their real-world applicability. To overcome these challenges, we have designed a lightweight, realtime architecture for classifying motions of the arm & hand using features derived from motor unit action potentials within surface Electromyographic (sEMG) signals, rather than which provide direct interrogation of underlying muscle activation patterns. We tested the architecture on 6 motions performed dynamically across a range of muscle contraction intensities achieving median classification accuracies ranging from 91.3% to 93.3 % and an average processing time of approximately 40 ms across three different classifiers. Taken together, our findings demonstrate potential robustness of motor unit based neural interfaces for motion classification tasks.

Teleoperation enables complex robot platforms to perform tasks beyond the scope of the current state-of-the-art robot autonomy by imparting human intelligence and critical thinking to these operations. For seamless control of robot platforms, it is essential to facilitate optimal situational awareness of the workspace for the operator through active telepresence cameras. However, the control of these active telepresence cameras adds an additional degree of complexity to the task of teleoperation. In this paper we present our results from the user study that investigates: (1) how the teleoperator learns or adapts to performing the tasks via active cameras modeled after camera placements on the TRINA humanoid robot; (2) the perception-action coupling operators implement to control active telepresence cameras, and (3) the camera preferences for performing the tasks. These findings from the human motion analysis and post-study survey will help us determine desired design features for robot teleoperation interfaces and assistive autonomy.

Robot teleoperation via human motion tracking has been proven to be easy to learn, intuitive to operate, and facilitate faster task execution than existing baselines. However, precise control while performing the dexterous telemanipulation tasks is still a challenge. In this paper, we implement sensory augmentation in terms of haptic and augmented reality visual cues to represent four types of information critical to the precision and performance of a telemanipulation task, namely: (1) target location; (2) constraint alert; (3) grasping affordance; and (4) grasp confirmation. We further conduct two user studies to investigate the effectiveness and preferred modality of the sensory feedback against no sensory support, and how the preference will be influenced by the different types of simulated real-world additional workload. We asked 8 participants to perform a general manipulation task using a KINOVA robotic arm. Our results indicate that: (1) the haptic and AR visual cues can significantly reduce the task completion time, occurrences of errors, the total length traversed by the robot end-effector, the operational effort while increasing the interface usability; (2) the haptic feedback trended in the direction of presenting the information that needs a prompt response, while the AR visual cues are suitable to monitor the system status; (3) the participants chose their preferred feedback with the purpose of reducing the cognitive workload despite increased extra effort.

Motion tracking interfaces are intuitive for free-form teleoperation tasks. However, efficient manipulation control can be difficult with such interfaces because of issues like the interference of unintended motions and the limited precision of human motion control. The limitation in control efficiency reduces the operator's performance and increases their workload and frustration during robot teleoperation. To improve the efficiency, we proposed separating controlled degrees of freedom (DoFs) and adjusting the motion scaling ratio of a motion tracking interface. The motion tracking of handheld controllers from a Virtual Reality system was used for the interface. We separated the translation and rotational control into: 1) two controllers held in the dominant and non-dominant hands and 2) hand pose tracking and trackpad inputs of a controller. We scaled the control mapping ratio based on 1) the environmental constraints and 2) the teleoperator's control speed. We further conducted a user study to investigate the effectiveness of the proposed methods in increasing efficiency. Our results show that the separation of position and orientation control into two controllers and the environment-based scaling methods perform better than their alternatives.

Tele-nursing robots provide a safe approach for patient-caring in quarantine areas. For effective nurse–robot collaboration, ergonomic teleoperation and intuitive interfaces with low physical and cognitive workload must be developed. We propose a framework to evaluate the control interfaces to iteratively develop an intuitive, efficient, and ergonomic teleoperation interface. The framework is a hierarchical procedure that incorporates general to specific assessment and its role in design evolution. We first present pre-defined objective and subjective metrics used to evaluate three representative contemporary teleoperation interfaces. The results indicate that teleoperation via human motion mapping outperforms the gamepad and stylus interfaces. The tradeoff with using motion mapping as a teleoperation interface is the non-trivial physical fatigue. To understand the impact of heavy physical demand during motion mapping teleoperation, we propose an objective assessment of physical workload in teleoperation using electromyography. We find that physical fatigue happens in the actions that involve precise manipulation and steady posture maintenance. We further implemented teleoperation assistance in the form of shared autonomy to eliminate the fatigue-causing component in robot teleoperation via motion mapping. The experimental results show that the autonomous feature effectively reduces the physical effort while improving the efficiency and accuracy of the teleoperation interface.

Robot teleoperation is a reliable way to perform a variety of tasks with complex robotic systems. However, the remote control of active telepresence cameras on the robot for improved telepresence adds an additional degree of complexity while teleoperating and can thus affect the operator’s performance during tele-manipulation. Our previous user study investigates the general human performance and preference when using various wearable cameras. In this paper, we further investigate how humans respond to the usage of telepresence cameras in terms of motion behavior. The findings from our human motion analysis inform several desired designs for robot teleoperation interfaces and assistive autonomy.

Supervisory control of a humanoid robot in a manipulation task requires coordination of remote perception with robot action, which becomes more demanding with multiple moving cameras available for task supervision. We explore the use of autonomous camera control and selection to reduce operator workload and improve task performance in a supervisory control task. We design a novel approach to autonomous camera selection and control, and evaluate the approach in a user study which revealed that autonomous camera control does improve task performance and operator experience, but autonomous camera selection requires further investigation to benefit the operator’s confidence and maintain trust in the robot autonomy.

Motion mapping is an intuitive method of teleoperation with a low learning curve. Our previous study investigates the physical fatigue caused by teleoperating a robot to perform general-purpose assistive tasks and this fatigue affects the operator’s performance. The results from that study indicate that physical fatigue happens more in the tasks which involve more precise manipulation and steady posture maintenance. In this paper, we investigate how teleoperation assistance in terms of shared autonomy can reduce the physical workload in robot teleoperation via motion mapping. Specifically, we conduct a user study to compare the muscle effort in teleoperating a mobile humanoid robot to (1) reach and grasp an individual object and (2) collect objects in a cluttered workspace with and without an autonomous grasping function that can be triggered manually by the teleoperator. We also compare the participants’ task performance, subjective user experience, and change in attitude towards the usage of teleoperation assistance in the future based on their experience using the assistance function. Our results show that: (1) teleoperation assistance like autonomous grasping can effectively reduce the physical effort, task completion time and number of errors; (2) based on their experience performing the tasks with and without assistance, the teleoperators reported that they would prefer to use automated functions for future teleoperation interfaces.

Telepresence tele-action robots enable human workers to reliably perform difficult tasks in remote, cluttered, and human environments. However, the effort to control coordinated manipulation and active perception motions may exhaust and intimidate novice workers. We hypothesize that such cognitive efforts would be effectively reduced if the teleoperators are provided with autonomous camera selection and control aligned with the natural perception-action coupling of the human motor system. Thus, we conducted a user study to investigate the coordination of active perception control and manipulation motions performed with visual feedback from various wearable and standalone cameras in a telepresence scenario. Our study discovered rich information about telepresence camera selection to inform telepresence system configuration and possible teleoperation assistance design for reduced cognitive effort in robot teleoperation.

High-level control is generally preferred for the control of complex robot platforms and by users inexperienced with robot teleoperation. However, high-level teleoperation interfaces can be less effective if the robot autonomy is not reliable. To address this problem, it is important to understand how the users' preference of teleoperation interface may vary with the reliability of the robot autonomy, and understand what design features ameliorate the frustration and effort caused by unreliable autonomy.This paper proposes a graphical user interface for high-level robot control. The framework of the interface enables teleoperators to control a robot at the action level, and incorporates a simple but effective design that enables teleoperators to recover from task failure in a number of ways. We conducted a user study (N = 25) to compare the performance and user experience when using the proposed high-level interface to a low-level interface (i.e., gamepad) for robot low-level control, on a representative manipulation task. We also investigated if the high-level teleoperation interface remains effective if the reliability of robot autonomy decreases. Our results show that a high-level interface able to handle the most frequent errors is resilient to the effects of unreliable robot autonomy. Although the total task completion time increased as the robot autonomy becomes unreliable, the users' perception of workload and task performance are not affected. Through the user study, we also reveal the desirable interface features.

It is important that automotive engineers understand the interactions between active human maneuvering motions and vehicle dynamics, and how vehicle control affects the physical sensations of the human driver. This paper proposes a new system framework, the human model-based active driving system (HuMADS) for simulating human driver-vehicle interactions. HuMADS integrates the vehicle controller with models of vehicle dynamics and human biomechanics. It has an hierarchical closed-loop architecture for driver-vehicle control systems, including structures and contact interfaces of human and vehicle bodies. HuMADS is based on the OpenSim simulation platform. The developed system regulates the human model dynamics, such that the human model can react realistically to vehicle maneuver motions. The usability of the HuMADS is demonstrated through the simulation of coordinated gas/brake pedal operation and wheel-steering in highway driving tasks. The simulated vehicle dynamics and vehicle maneuvers are comparable with previously published experimental data of car-following driving. In addition, the proposed controllers successfully maintain the human body's balance inside the vehicle during vehicle maneuvers. We are convinced that the HuMADS has potential as a tool for the development of intelligent transportation systems and investigation of integrated safety.

Handing over objects is the foundation of many human-robot interaction and collaboration tasks. In the scenario where a human is handing over an object to a robot, the human chooses where the object needs to be transferred. The robot needs to accurately predict this point of transfer to reach out proactively, instead of waiting for the final position to be presented. This work presents an efficient method for predicting the Object Transfer Point (OTP), which synthesizes (1) an offline OTP calculated based on human preferences observed in a human-robot motion study with (2) a dynamic OTP predicted based on the observed human motion. Our proposed OTP predictor is implemented on a humanoid nursing robot and experimentally validated in human-robot handover tasks. Compared to only using static or dynamic OTP estimators, it has better accuracy at the earlier phase of handover (up to 45% of the handover motion) and can render fluent handovers with a reach-to-grasp response time (about 3.1 secs) close to natural human receiver's response. In addition, the OTP prediction accuracy is maintained across the robot's visible workspace by utilizing a user-adaptive reference frame.

Robot teleoperation via motion mapping has been demonstrated to be an efficient and intuitive approach for controlling and teaching the whole-body motion coordination of humanoid robots. However, the physical fatigue in the usage of such robot teleoperation interfaces may prevent this approach to be widely used in large scale by diverse workforce populations. As a result, this paper conducts a user study to investigate the physical fatigue of teleoperators in the whole-body motion mapping teleoperation of a mobile humanoid assistive robot. Through a Vicon motion capture system, participants teleoperated the robot to perform general purpose assistive tasks that involve reaching-to-grasp, bimanual manipulation, loco-manipulation and human-robot interaction. We assess the physical fatigue based on surface electromyography (sEMG) measurement, and compare it between different tasks and muscles. Our analysis results indicate that: (1) Fatigue happens more in the tasks that involve more precise manipulation and steady posture maintenance; (2) Deltoids, Biceps and Trapezius are used more for such tasks and thus have more fatigue than others. These findings imply that automating the fatigue-causing task components may reduce the physical fatigue in motion mapping teleoperation.

High-fidelity computational human models provide a safe and cost-efficient method for studying driver experience in vehicle maneuvers and for validation of vehicle design. Compared to passive human models, active human models capable of reproducing the decision-making, as well as vehicle maneuver motion planning and control, will be able to support realistic simulation of human-vehicle interaction. In this paper, we propose an integrated human-vehicle interaction simulation framework which learns vehicle maneuver motion primitives from human drivers, and uses them to compose natural and contextual driving motions. Specifically, we recruited six experienced drivers and recorded their vehicle maneuver motions on a fixed-base driving simulation testbed. We further segmented and classified the collected data based on their similarity in joint coordination. Using a combination of imitation learning methods, we extracted the regularity and variability of vehicle maneuver motions across subjects, and learned the dynamic motion primitives to be used for motion reproduction in simulation. We present an implementation of the framework on lower-extremity joint coordination in pedal activation for longitudinal vehicle control. Our research efforts lead to a motion primitive library which enables planning natural driver motions, and will be integrated with the driving decision-making, motion control, and vehicle dynamics in the proposed framework for simulating human-vehicle interaction.

During outbreaks of contagious diseases, healthcare workers are at high risk for infection due to routine interaction with patients, handling of contaminated materials, and challenges associated with safely removing protective gear. This poses an opportunity for the use of remote-controlled robots that could perform common nursing duties inside hazardous clinical areas, thereby minimizing the exposure of healthcare workers to contagions and other biohazards. This paper describes the development of the prototype system Tele-Robotic Intelligent Nursing Assistant (TRINA), which consists of a mobile manipulator robot, a human operator's console, and operator assistance algorithms which automate or partially-automate tedious and error-prone tasks. Using off-the-shelf robotic and sensing components, total hardware costs are kept under $75,000. The system's capabilities for performing standard nursing tasks are evaluated in the simulation laboratory of a nursing school.

The addition of manipulation capabilities to telepresence robots holds the promise of enabling remote humans to perform tele-labor, hands-on training, and collaborative manipulation, but the use of a robot as a mediator to humanhuman physical interaction is not yet well understood. This paper studies the impact of telepresence modalities in the context of robot-mediated object handover. A teleoperation system was developed involving a bimanual mobile manipulator with telepresence head and sensing capabilities, and a user study was conducted with n=10 pairs of subjects under a variety of audio and visual telepresence conditions. Results show that telepresence does not significantly affect objective handover fluency, but both audio and video telepresence do significantly improve user experience on subjective measures including intimacy and perceived fluency.

Despite increasing popularity of compliant and underactuated hands, few tools are available for modeling them. Thus, we propose a simulation technique to predict the success of a compliant gripper grasping irregular objects, which could be used in mechanism design as well as grasp planning. The simulator we propose integrates joint compliance simulation with a Boundary Layer Expanded Mesh (BLEM) technique to enhance the stability of contact estimation. We compare the proposed simulator with existing simulators via a set of stability and fidelity criteria, including contact force variation, contact position variation, and contact normal variation. Scores along these criteria are correlated with the simulator's accuracy of predicting the success/failure of a given grasp pose and preshape. A test set of 13 grasps, with two compliant underactuated hands were manually generated on 4 objects. Experiments suggest that our simulator leads to improvements in the stability criteria, predictability of grasp success, and reduction of simulation artifacts.

Reach-to-grasp movements are widely observed in activities of daily living, particularly in tool manipulations. In order to reduce the complexity in redundancy resolution and facilitate upper-limb exoskeleton control in reach-to-grasp tasks, we studied joint coordination in the human arm during such movements. Experimental data were collected on reach-to-grasp movements in a 3-dimensional (3D) workspace for cylinder targets of different positions and grasping orientations. For comparison, reaching movements toward the same targets are also recorded. In the kinematic analysis, the redundant degree of freedom in human arm is represented by the swivel angle. The four grasping-relevant degrees of freedom (GR-DOFs), including the swivel angle and the three wrist joints, behave differently in reach-to-grasp movements comparing to how they behave in reaching movements. The ratio of active motion range (R-AMR) is proposed for quantitatively comparison the task-relevance of the GR-DOFs. Analysis on the R-AMR values shows that the task-relevant GR-DOFs are more actively used, while the task-irrelevant joints are left uncontrolled and maintain their neutral positions. Among the task-relevant GR-DOFs, the smaller joints (micro-structure) are more actively used than the larger joints (macro-structure). The coordination of the task-relevant GR-DOFs is shown to be synergistic. Analysis of the acceleration/deceleration at the GR-DOFs indicates different levels of voluntary control in three phases of the movements. The study of the characteristics of the joint coordination in reach-to-grasp movements provides guide-lines for simplifying the control of the upper limb exoskeleton.

According to the seven degrees of freedom (DOFs) human arm model composed of the shoulder, elbow, and wrist joints, positioning of the wrist in space and orientating the palm is a task requiring only six DOFs. Due to this redundancy, a given task can be completed by multiple arm configurations, and there is no unique mathematical solution to the inverse kinematics. The redundancy of a wearable robotic system (exoskeleton) that interacts with the human is expected to be resolved in the same way as that of the human arm. A unique solution to the system's redundancy was introduced by combining both kinematic and dynamic criteria. The redundancy of the arm is expressed mathematically by defining the swivel angle: the rotation angle of the plane including the upper and lower arm around a virtual axis connecting the shoulder and wrist joints which are fixed in space. Two different swivel angles were generated based on kinematic and dynamic constraints. The kinematic criterion is to maximize the projection of the longest principle axis of the manipulability ellipsoid for the human arm on the vector connecting the wrist and the virtual target on the head region. The dynamic criterion is to minimize the mechanical work done in the joint space for each two consecutive points along the task space trajectory. These two criteria were then combined linearly with different weight factors for estimating the swivel angle. Post processing of experimental data collected with a motion capturing system indicated that by using the proposed synthesis of redundancy resolution criteria, the error between the predicted swivel angle and the actual swivel angle adopted by the motor control system was less then five degrees. This result outperformed the prediction based on a single criteria.

Surgical procedures are traditionally performed by two or more surgeons along with staff nurses. One surgeon serves as the primary surgeon and the other serves as his/her assistant. Surgical robotics have redefined the dynamics in which the two surgeons interact with each other and with the surgical site. Raven IV is a new generation of the surgical robot system having four articulated robotic arms in a spherical configuration, each holding an articulated surgical tool. The system allows two surgeons to teleoperate the Raven IV collaboratively from two remote sites. The current research effort aims to configure the link architecture of each robotic arm, along with the position (port placement) and orientation of the Raven IV with respect to the patient, in order to optimize the common workspace reachable by all four robotic arms. The simulation results indicate that tilting the base of the robotic arms in the range of -20 to 20 deg while moving the ports closer together up to 50 mm apart leads to a preferred circular shape of the common workspace with an isotropy value of 0.5. A carefully configured system with multiple surgical robotic arms will enhance the interactive performance of the two surgeons.

Networked haptic cooperation entails direct interaction among users as well as joint manipulation of virtual objects. To increase the realism of both types of interactions, this paper introduces remote dynamic proxies. Remote dynamic proxies are second order dynamic representations of users at the remote peer sites. They are generated according to dynamics laws and are controlled by the user whom they represent through a virtual coupler. Hence, they move in a physically intuitive manner and do not suffer from position discontinuities due to network packet transmission limitations. The remote dynamic proxies are integrated into a distributed control architecture for networked haptic cooperation. An experimental comparison of the new controller to two recently proposed controllers demonstrates smoother rendering of contact between users, as well as stable cooperation for larger network delays.