We address the problem of intention estimation in human-robot teleoperation, which involves identifying the task being completed and predicting the next actions. Our approach sequentially quantifies the similarity between the observed action sequence and nominal action sequences representing possible tasks using the edit distance metric. Task estimation and action prediction are then performed using a nearest-neighbor rule. A key advantage of our approach is its robustness to deviations in operator actions and action recognition errors, commonly encountered in real-world teleoperation settings. Through extensive experiments on both real and simulated data, we demonstrate that our method largely outperforms alternative approaches, including probabilistic graphical models and transformer-based methods, particularly in scenarios with significant action deviations or action recognition errors. Additionally, we construct task distance matrices to analyze task similarities and potential confusion points, providing insights into when and where estimation errors are likely to occur. This analysis can guide the design of more distinctive task sequences and further improve the reliability of teleoperated robotic systems.

We propose a new approach to solving the problem of intention estimation in human-robot teleoperation for assembly tasks, which includes task estimation and action prediction. Our approach uses probabilistic graphical models to represent the joint distribution of the task and the actions to be taken to complete the task. Both model learning and inference are implemented with Pyro, a state-of-the-art probabilistic programming language. The distinctive feature from the traditional hidden Markov model type of probabilistic methods is that our model takes the time information into account and explicitly models the individual distributions of all the variables under consideration. By doing this, we fully utilize the power of probabilistic programming, and achieve accurate distribution hence uncertainty estimations. Working with a pretrained action recognition module, the proposed model can be trained solely on a tiny instruction manual of the assembly tasks and can be retrained with minimal overhead whenever the manual is changed or augmented, avoiding the need for the costly data reannotation and retraining by the end-to-end learning based methods. We also compare our method with a transformer based model trained directly on the instruction manual, and our method shows superior accuracy in both intention estimation and their distribution estimations. We additionally identify failure cases of both our method and the transformer-based method, and envision methods for improvement.

The driver warning system that alerts the human driver about potential risks during driving is a key feature of an advanced driver assistance system. Existing driver warning technologies, mainly the forward collision warning and unsafe lane change warning, can reduce the risk of collision caused by human errors. However, the current design methods have several major limitations. Firstly, the warnings are mainly generated in a one-shot manner without modeling the ego driver’s reactions and surrounding objects, which reduces the flexibility and generality of the system over different scenarios. Additionally, the triggering conditions of warning are mostly rule-based threshold-checking given the current state, which lacks the prediction of the potential risk in a sufficiently long future horizon. In this work, we study the problem of optimally generating driver warnings by considering the interactions among the generated warning, the driver behavior, and the states of ego and surrounding vehicles on a long horizon. The warning generation problem is formulated as a partially observed Markov decision process (POMDP). An optimal warning generation framework is proposed as a solution to the proposed POMDP. The simulation experiments demonstrate the superiority of the proposed solution to the existing warning generation methods.