Simulation to Real-World Transfer allows affordable and fast training of learning-based robots for manipulation tasks using Deep Reinforcement Learning methods. Currently, Asymmetric Actor-Critic approaches are used for Sim2Real to reduce the rich idealized features in simulation to the accessible ones in the real world. However, the feature reduction from the simulation to the real world is conducted through an empirically defined one-step curtail. Small feature reduction does not sufficiently remove the actor’s features, which may still cause difficulty setting up the physical system, while large feature reduction may cause difficulty and inefficiency in training. To address this issue, we proposed Curriculum-based Sensing Reduction to enable the actor to start with the same rich feature space as the critic and then get rid of the hard-to-extract features step-by-step for higher training performance and better adaptation for real-world feature space. The reduced features are replaced with random signals from a Deep Random Generator to remove the dependency between the output and the removed features and avoid creating new dependencies. The methods are evaluated on the Allegro robot hand in a real-world in-hand manipulation task. The results show that our methods have faster training and higher task performance than baselines and can solve real-world tasks when selected tactile features are reduced.

Dexterous telemanipulation is crucial in advancing human-robot systems, especially in tasks requiring precise and safe manipulation. However, it faces significant challenges due to the physical differences between human and robotic hands, the dynamic interaction with objects, and the indirect control and perception of the remote environment. Current approaches predominantly focus on mapping the human hand onto robotic counterparts to replicate motions, which exhibits a critical oversight: it often neglects the physical interaction with objects and relegates the interaction burden to the human to adapt and make laborious adjustments in response to the indirect and counter-intuitive observation of the remote environment. This work develops an End-Effects-Oriented Learning-based Dexterous Telemanipulation (EFOLD) framework to address telemanipulation tasks. EFOLD models telemanipulation as a Markov Game, introducing multiple end-effect features to interpret the human operator’s commands during interaction with objects. These features are used by a Deep Reinforcement Learning policy to control the robot and reproduce such end effects. EFOLD was evaluated with real human subjects and two end-effect extraction methods for controlling a virtual Shadow Robot Hand in telemanipulation tasks. EFOLD achieved real-time control capability with low command following latency (delay<0.11s) and highly accurate tracking (MSE<0.084 rad).

Filter-based shared control aims to accept and augment an operator's ability to control a robot. Current solutions accept actions based on their direction aligning with the robot's optimal policy. These strategies reject a human's small corrective actions if they conflict with the robot's direction and accept too aggressive actions as long as they are consistent with the robot's direction. Such strategies may cause task failures and the operator's feeling of loss of control. To close the gap, we propose WE-Filter, which has flexible, adaptive criteria allowing the operator's small corrective actions and tempering too aggressive ones. Inspired by classical work-energy impact problems between two dynamic, interactive bodies, both inputs' properties (direction and magnitude) are inherently considered, creating intuitive, adaptive bounds to accept sensible actions. The model identifies behaviors before and after impact. The rationale is that each timestep of shared control acts as an impact between the operator's and the robot's policies, where post-impact behaviors depend on their previous behaviors. As time continues, a series of impacts occur. The aim is to minimize impacts that occur to reach an agreement faster and reduce strong reactionary behaviors. Our model determines flexible acceptance criteria to bound a mismatch of magnitude and finds a replacement action for conflicting policies. The WE-Filter achieves better task performance, the ratio of accepted actions, and action similarity than the existing methods.

In-hand manipulation is challenging for a multi-finger robotic hand due to its high degrees of freedom and complex interaction with the object. To enable in-hand manipulation, existing deep reinforcement learning-based approaches mainly focus on training a single robot-structure-specific policy through the centralized learning mechanism, lacking adaptability to changes like robot malfunction. To solve this limitation, this work treats each finger as an individual agent and trains multiple agents to control their assigned fingers to complete the in-hand manipulation task cooperatively. We propose the Multi-Agent Global-Observation Critic and Local-Observation Actor (MAGCLA) method, where the critic can observe all agents' actions globally, and the actor only locally observes its neighbors' actions. Besides, conventional individual experience replay may cause unstable cooperation due to the asynchronous performance increment of each agent, which is critical for in-hand manipulation tasks. To solve this issue, we propose the Synchronized Hindsight Experience Replay (SHER) method to synchronize and efficiently reuse the replayed experience across all agents. The methods are evaluated in two in-hand manipulation tasks on the Shadow dexterous hand. The results show that SHER helps MAGCLA achieve comparable learning efficiency to a single policy, and the MAGCLA approach is more generalizable in different tasks. The trained policies have higher adaptability in the robot malfunction test compared to the baseline multi-agent and single-agent approaches.

While a human is tracking a moving object to prepare for later grasping, we naturally change our hand pose to generate optimal pre-grasp to avoid post-grasp adjustment. Robot hand controllers need dynamic pre-grasp planning capability, so they are not limited in dynamic tracking and catching tasks. To fill this gap, we explore the feasibility of using a two-stage optimization method to enable dynamic pre-grasp planning of individual fingers while tracking a moving object to ensure a later successful grasp. The first stage adopts multi-agent pursuit to partition the search space on the object surface. The method allows each finger to consider its immediate surroundings in a local view instead of globally determining the best location for all fingers. The search space for each finger is dramatically reduced since sensible alternatives are the ones left after pruning. Each finger goal location acts independently yet coordinates with others to achieve the goal of covering the object. In the second stage, four different goal point movement strategies are presented to impact the finger goal location in their respective search space to demonstrate the ability to facilitate different needs of the task and requirements of the designer. Dynamic finger goal adaption is obtained by iteratively updating these two stages. The approach is consistent in different scenarios for the object.

Appropriately allocating the operator’s attention for single-operator-multi-robot systems is critical yet still an open problem. Failing to effectively allocate the operator’s attention may cause low situational awareness and wasted human attention on nonessential events, which leads to decreased team performance. In this paper, a novel active attention allocation strategy based on robot performance prediction is developed. Different from previous work, this strategy attempts to avoid potential issues before their occurrence instead of fixing them afterward. Based on a mathematical robot performance model, a robot’s performance level can be predicted with accumulated human attention given to that robot. The human attention can then be actively allocated to maximize team performance in the short future. Simulations using a domain search task are performed. The results demonstrate the validity of the attention-based robot performance model and the effectiveness of the active attention allocation strategy in improving team performance.

Applying Deep Reinforcement Learning (DRL) to Human-Robot Cooperation (HRC) in dynamic control problems is promising yet challenging as the robot needs to learn the dynamics of the controlled system and dynamics of the human partner. In existing research, the robot powered by DRL adopts coupled observation of the environment and the human partner to learn both dynamics simultaneously. However, such a learning strategy is limited in terms of learning efficiency and team performance. This work proposes a novel task decomposition method with a hierarchical reward mechanism that enables the robot to learn the hierarchical dynamic control task separately from learning the human partner’s behavior. The method is validated with a hierarchical control task in a simulated environment with human subject experiments. Our method also provides insight into the design of the learning strategy for HRC. The results show that the robot should learn the task first to achieve higher team performance and learn the human first to achieve higher learning efficiency.

Promoting a robot agent's autonomy level, which allows it to understand the human operator's intent and provide motion assistance to achieve it, has demonstrated great advantages to the operator's intent in teleoperation. However, the research has been limited to the target approaching process. We advance the shared control technique one step further to deal with the more challenging object manipulation task. Appropriately manipulating an object is challenging as it requires fine motion constraints for a certain manipulation task. Although these motion constraints are critical for task success, they are subtle to observe from ambiguous human motion. The disembodiment problem and physical discrepancy between the human and robot hands bring additional uncertainty, make the object manipulation task more challenging. Moreover, there is a lack of modeling and planning techniques that can effectively combine the human motion input and robot agent's motion input while accounting for the ambiguity of the human intent. To overcome this challenge, we built a multi-task robot grasping model and developed an intent-uncertainty-aware grasp planner to generate robust grasp poses given the ambiguous human intent inference inputs. With this validated modeling and planning techniques, it is expected to extend teleoperated robots' functionality and adoption in practical telemanipulation scenarios.

The emergence of assistive robots presents the possibility of restoring vital degrees of independence to the elderly and impaired in activities of daily living (ADL). However, one of the main challenges is the lack of a means for effective and intuitive human-robot interaction (HRI). While humans can express their intentions in different ways (e.g., physical gestures or motions, or speech or language patterns), gaze-based implicit intention communication is still underdeveloped. In this study, a novel nonverbal implicit communication framework based on eye gaze is introduced for HRI. In this framework, a user's eye-gaze movements are proactively tracked and analyzed to infer the user's intention in ADL. Then, the inferred intention can be used to command assistive robots for proper service. The advantage of this framework is that gaze-based communication can be handled by most of the people, as it requires very little effort, and most of the elderly and impaired retain visual capability. This framework is expected to simplify HRI, consequently enhancing the adoption of assistive technologies and improving users' independence in daily living. The testing results of this framework confirmed that a human's subtle gaze cues on visualized objects could be effectively used for human-intention communication. Results also demonstrated that the gaze-based intention communication is easy to learn and use. In this study, the relationship of visual behaviors with the mental process during human intention expression was studied for the first time to build a fundamental understanding of this process. These findings are expected to guide further design of accurate intention inference algorithms and intuitive HRI.

To provide the elderly with appropriate and timely caregiving in activities of daily life (ADL), it is desired for robots to have the capability of intention awareness (IA). Different from existing context-specific intention awareness (CSIA) approaches which are based on a limited and passive knowledge database, our approach, named `WebIA', adopts a novel web query approach to establish a vast and active knowledge database for robots to perform CSIA. In our method, robots are endowed with comprehensive commonsense knowledge towards the correlations of the surrounding environment and human intentions. WebIA enables robots to effectively infer intentions in both trained familiar and untrained new situations. By performing several experiments, we evaluated two aspects of the WebIA approach: the effectiveness of IA in familiar situations and the self-learning ability in new situations.