Manually teleoperating a flying robot can be a demanding task, especially for users with limited levels of experience. This is primarily due to the non-linear properties of such robots in addition to the difficulty of controlling various degrees of freedom at the same time. 3D Autocomplete helps mitigate such limitations by assisting the users in teleoperation. It aids in teleoperating 3D motions, such as helical motions, which are more challenging to the users. The proposed framework uses Artificial Intelligence (AI) to predict just-in-time the user’s intended motion and then, if the user accepts, completes it autonomously in 3D. The AI component of 3D Autocomplete was presented in our previous work, where we introduced a deep learning model and an algorithm to predict as early as possible the user’s desired motion. Moving forward in this work, we focus on synthesizing and completing the user-intended motion autonomously. Also, we introduce a Mixed Reality (MR) user interface for better human-robot interaction. Finally, we evaluate our system subjectively and objectively through human-subject experiments. Autocomplete outperformed traditional method on all criteria with at least 30% improvement in all objective measures.

Tele-operating aerial vehicles without any automated assistance is challenging due to various limitations, especially for inexperienced users. Autocomplete addresses this problem by automatically identifying and completing the user's intended motion. Such a framework uses machine learning to recognize and classify human inputs as one of a set of motion primitives, and then, if the human operator accepts, synthesizes the motion in order to complete the desired motion. This has been shown to improve the performance of the system and reduce operator workload. Previous Autocomplete systems focused on different 2D motions (line, arc, sine,..). However, since most UAVs tasks are in a 3D world, this paper introduces 3D Autocomplete for 3D motions. Moreover, the proposed framework presents just-in-time prediction of the 3D motions by proposing a change point detection technique, which allows the framework to autonomously identify when to conduct a prediction. Also, it deals with variable motion sizes. Real time simulation results show that the proposed framework is capable of predicting the user intentions after change point detection.

Remote controlling robots without any automated help is difficult due to various limitations. Autocomplete mitigates this difficulty by automatically detecting and completing the intended motions on robots from the input of the user. Such an approach can improve the system performance and reduce the load on the operator. Usually, recognizing intended motions is achieved using pre-trained Deep Learning (DL) models. In this paper, we introduce personalization to the autocomplete teleoperation framework when new operators take over by customizing the autocomplete DL model using incremental learning. Also, we tackle the problem of concept drift that arises in real-life applications; the data distribution of already learned classes may change in unforeseen ways as new observations of these classes come sequentially over time. We create and update an exemplar set using new observations of the classes online so that the model can be trained to adapt to the new observations. Several scenarios have been evaluated to balance the speed of learning with the accuracy of the model, and results demonstrate the effectiveness of the proposed models and their advantage in adapting to the specific operator versus our previous framework: personalization using transfer learning with full feedback.