Shared autonomy can improve teleoperating robotic systems in complex manufacturing and assembly tasks by combining human decision-making and robotic capabilities. A key aspect of seamless collaboration and trust in shared autonomy is the robot’s ability to interpret human intentions in a consistent and explainable manner. To achieve this, a graph neural network-based intention estimation framework is introduced, which generates dynamic graphs that capture spatial relationships evolving over time. The framework predicts human intentions at two hierarchical levels: low-level actions and high-level tasks. Furthermore, we empirically and anecdo-tally verify the correctness and consistency of the predictions using explainability metrics. The algorithm is demonstrated by teleoperating a bi-manual robot to assemble various block structures in a virtual reality simulation environment.

We address the problem of intention estimation in human-robot teleoperation, which involves identifying the task being completed and predicting the next actions. Our approach sequentially quantifies the similarity between the observed action sequence and nominal action sequences representing possible tasks using the edit distance metric. Task estimation and action prediction are then performed using a nearest-neighbor rule. A key advantage of our approach is its robustness to deviations in operator actions and action recognition errors, commonly encountered in real-world teleoperation settings. Through extensive experiments on both real and simulated data, we demonstrate that our method largely outperforms alternative approaches, including probabilistic graphical models and transformer-based methods, particularly in scenarios with significant action deviations or action recognition errors. Additionally, we construct task distance matrices to analyze task similarities and potential confusion points, providing insights into when and where estimation errors are likely to occur. This analysis can guide the design of more distinctive task sequences and further improve the reliability of teleoperated robotic systems.

We propose a new approach to solving the problem of intention estimation in human-robot teleoperation for assembly tasks, which includes task estimation and action prediction. Our approach uses probabilistic graphical models to represent the joint distribution of the task and the actions to be taken to complete the task. Both model learning and inference are implemented with Pyro, a state-of-the-art probabilistic programming language. The distinctive feature from the traditional hidden Markov model type of probabilistic methods is that our model takes the time information into account and explicitly models the individual distributions of all the variables under consideration. By doing this, we fully utilize the power of probabilistic programming, and achieve accurate distribution hence uncertainty estimations. Working with a pretrained action recognition module, the proposed model can be trained solely on a tiny instruction manual of the assembly tasks and can be retrained with minimal overhead whenever the manual is changed or augmented, avoiding the need for the costly data reannotation and retraining by the end-to-end learning based methods. We also compare our method with a transformer based model trained directly on the instruction manual, and our method shows superior accuracy in both intention estimation and their distribution estimations. We additionally identify failure cases of both our method and the transformer-based method, and envision methods for improvement.

In human-robot collaboration, shared control presents an opportunity to teleoperate robotic manipulation to improve the efficiency of manufacturing and assembly processes. Robots are expected to assist in executing the user’s intentions. To this end, robust and prompt intention estimation is needed, relying on behavioral observations. The framework presents an intention estimation technique at hierarchical levels i.e., low-level actions and high-level tasks, by incorporating multi-scale hierarchical information in neural networks. Technically, we employ hierarchical dependency loss to boost overall accuracy. Furthermore, we propose a multi-window method that assigns proper hierarchical prediction windows of input data. An analysis of the predictive power with various inputs demonstrates the predominance of the deep hierarchical model in the sense of prediction accuracy and early intention identification. We implement the algorithm on a virtual reality (VR) setup to teleoperate robotic hands in a simulation with various assembly tasks to show the effectiveness of online estimation. Video demonstration is available at: https://youtu.be/CMYDgcI4j1g.

The need to design for appropriate human-autonomy teaming has become increasingly important as systems grow in complexity, especially those that require time-pressured interactions like in unmanned aerial vehicle (UAV) operations. However, it is not always clear whether operators develop effective strategies for computer-based technologies. When operators are given such tools, their performances can be statistically compared but often such assessments only provide summative information. These comparisons do not indicate how and why technology influenced people's strategies and actions. To fill this gap, we utilized Hidden Markov Models (HMMs) to represent strategies employed by operators in first-person control of UAVs for inspection tasks. The resulting models captured differences in strategies for people who both succeeded and crashed, as well as those who were overconfident in their self-assessments, and those who were not. People who were not overconfident exhibited less risky strategies and were more successful. These findings were further strengthened by a quantitative state similarity metric, which indicated where and for who possible interventions could improve outcomes. This application of HMMs to operator strategy representation could help to identify effective operator strategy development in the use of computer-based technologies, and what kind of interventions could be the most effective in improving outcomes.

Appropriately allocating the operator’s attention for single-operator-multi-robot systems is critical yet still an open problem. Failing to effectively allocate the operator’s attention may cause low situational awareness and wasted human attention on nonessential events, which leads to decreased team performance. In this paper, a novel active attention allocation strategy based on robot performance prediction is developed. Different from previous work, this strategy attempts to avoid potential issues before their occurrence instead of fixing them afterward. Based on a mathematical robot performance model, a robot’s performance level can be predicted with accumulated human attention given to that robot. The human attention can then be actively allocated to maximize team performance in the short future. Simulations using a domain search task are performed. The results demonstrate the validity of the attention-based robot performance model and the effectiveness of the active attention allocation strategy in improving team performance.

Promoting a robot agent's autonomy level, which allows it to understand the human operator's intent and provide motion assistance to achieve it, has demonstrated great advantages to the operator's intent in teleoperation. However, the research has been limited to the target approaching process. We advance the shared control technique one step further to deal with the more challenging object manipulation task. Appropriately manipulating an object is challenging as it requires fine motion constraints for a certain manipulation task. Although these motion constraints are critical for task success, they are subtle to observe from ambiguous human motion. The disembodiment problem and physical discrepancy between the human and robot hands bring additional uncertainty, make the object manipulation task more challenging. Moreover, there is a lack of modeling and planning techniques that can effectively combine the human motion input and robot agent's motion input while accounting for the ambiguity of the human intent. To overcome this challenge, we built a multi-task robot grasping model and developed an intent-uncertainty-aware grasp planner to generate robust grasp poses given the ambiguous human intent inference inputs. With this validated modeling and planning techniques, it is expected to extend teleoperated robots' functionality and adoption in practical telemanipulation scenarios.

The emergence of assistive robots presents the possibility of restoring vital degrees of independence to the elderly and impaired in activities of daily living (ADL). However, one of the main challenges is the lack of a means for effective and intuitive human-robot interaction (HRI). While humans can express their intentions in different ways (e.g., physical gestures or motions, or speech or language patterns), gaze-based implicit intention communication is still underdeveloped. In this study, a novel nonverbal implicit communication framework based on eye gaze is introduced for HRI. In this framework, a user's eye-gaze movements are proactively tracked and analyzed to infer the user's intention in ADL. Then, the inferred intention can be used to command assistive robots for proper service. The advantage of this framework is that gaze-based communication can be handled by most of the people, as it requires very little effort, and most of the elderly and impaired retain visual capability. This framework is expected to simplify HRI, consequently enhancing the adoption of assistive technologies and improving users' independence in daily living. The testing results of this framework confirmed that a human's subtle gaze cues on visualized objects could be effectively used for human-intention communication. Results also demonstrated that the gaze-based intention communication is easy to learn and use. In this study, the relationship of visual behaviors with the mental process during human intention expression was studied for the first time to build a fundamental understanding of this process. These findings are expected to guide further design of accurate intention inference algorithms and intuitive HRI.

To provide the elderly with appropriate and timely caregiving in activities of daily life (ADL), it is desired for robots to have the capability of intention awareness (IA). Different from existing context-specific intention awareness (CSIA) approaches which are based on a limited and passive knowledge database, our approach, named `WebIA', adopts a novel web query approach to establish a vast and active knowledge database for robots to perform CSIA. In our method, robots are endowed with comprehensive commonsense knowledge towards the correlations of the surrounding environment and human intentions. WebIA enables robots to effectively infer intentions in both trained familiar and untrained new situations. By performing several experiments, we evaluated two aspects of the WebIA approach: the effectiveness of IA in familiar situations and the self-learning ability in new situations.