With the aging of society, assistive companion robots have become an important solution to address the daily needs of the elderly. However, the elderly may suffer from slow thinking and memory loss, which makes it difficult for robots to accurately capture executable intentions during human-robot interaction. To address this challenge, this paper proposes an Adaptive Multimodal Complementary Intent Understanding Approach (AMCIU). First, a self-attentive mechanism is utilized to extract multimodal features of gestures, speech and images. Second, knowledge graph is utilized for the first time to obtain complementary information between image and audio modalities. Finally, robust intent integration is achieved through a multimodal hybrid expert intent fusion technique. In comparison with multiple state-of-the-art multimodal learning methods, AMCIU shows significant performance improvement in the building block tower scenario, where the intention understanding accuracy reaches 96.11%. This paper provides a novel research idea for natural human-robot interaction.

During the COVID-19 pandemic, online classes became the only option for many students. The main challenge for these classes was conducting risky and complex chemical or biological experiments in a domestic environment. To address this challenge, a smart experiment system called MRLab was developed. MRLab used wearables such as a smart glove and head-mounted device to record sensory data and a multimodal hybrid fusion model GVVS to interpret the user’s experimental intent, which essentially transforms the user’s abstract behavioral actions into a probabilistic set of experimental intent that can be computed. Different experiments in MRLab used different libraries of experimental intents. The SrNet model in GVVS was used to estimate the probability of the user’s gesture behavior generated from the smart glove, while the SIPA algorithm compared speech information entered during the experiment with the experimental intent library to estimate the probability of the user’s intent. At the same time, the scene visual channel monitored the information about the object the user intended to operate, with the SVF algorithm computing the probability of the intended object in real-time. The results from ANOVA and post-hoc comparative testing conducted on 21 volunteers revealed that MRLab outperformed other experiment modes, including WEB, AR, and VR, with a higher intention understanding rate, efficiency, and user satisfaction. Therefore, MRLab proved to be a useful alternative to traditional physics laboratory experiments during the pandemic, along with being an additional teaching tool for remote learning purposes.

The naturalness and safety of human-computer interaction have always been primary research focuses in the field of human-computer interaction. This paper proposes a multimodal intention understanding algorithm (MIUIC), which incorporates comfort analysis, as a solution to address the issues of low intention understanding rate, weak interaction, and weak collaboration that are often observed in most massage systems. The algorithm efficiently fuses multimodal data based on objective implicit information to address the challenge of low intention understanding rates caused by non-standard expression of natural behavior. Moreover, this algorithm incorporates comfort reasoning to detect and address intentions related to security threats while providing the ability for robots to make behavioral decisions through inverse active interaction, leading to more equitable human-robot interactions. To test the validity and safety of the MIUIC algorithm, we embedded the algorithm into a mechanical arm massage system. Subsequently, 45 elderly volunteers were invited to participate in experimental tests. Finally, to verify the validity and safety of the MIUIC algorithm, we assessed the algorithm in terms of four aspects, including multimodal intention recognition rate, the ability to reduce data dispersion, the intention enhancement rate under reverse human-machine interaction, and the rate of avoiding dangerous intentions. In conclusion, the MIUIC algorithm enhances the intention understanding rate and promotes.

At present, most experimental teaching systems lack guidance of an operator, and thus users often do not know what to do during an experiment. The user load is therefore increased, and the learning efficiency of the students is decreased. To solve the problem of insufficient system interactivity and guidance, an experimental navigation system based on multi-mode fusion is proposed in this paper. The system first obtains user information by sensing the hardware devices, intelligently perceives the user intention and progress of the experiment according to the information acquired, and finally carries out a multi-modal intelligent navigation process for users. As an innovative aspect of this study, an intelligent multi-mode navigation system is used to guide users in conducting experiments, thereby reducing the user load and enabling the users to effectively complete their experiments. The results prove that this system can guide users in completing their experiments, and can effectively reduce the user load during the interaction process and improve the efficiency.

Augmented reality classrooms have become an interesting research topic in the field of education, but there are some limitations. Firstly, most researchers use cards to operate experiments, and a large number of cards cause difficulty and inconvenience for users. Secondly, most users conduct experiments only in the visual modal, and such single-modal interaction greatly reduces the users' real sense of interaction. In order to solve these problems, we propose the Multimodal Interaction Algorithm based on Augmented Reality (ARGEV), which is based on visual and tactile feedback in Augmented Reality. In addition, we design a Virtual and Real Fusion Interactive Tool Suite (VRFITS) with gesture recognition and intelligent equipment. The ARGVE method fuses gesture, intelligent equipment, and virtual models. We use a gesture recognition model trained by a convolutional neural network to recognize the gestures in AR, and to trigger a vibration feedback after a recognizing a fivefinger grasp gesture. We establish a coordinate mapping relationship between real hands and the virtual model to achieve the fusion of gestures and the virtual model. The average accuracy rate of gesture recognition was 99.04%. We verify and apply VRFITS in the Augmented Reality Chemistry Lab (ARCL), and the overall operation load of ARCL is thus reduced by 29.42%, in comparison to traditional simulation virtual experiments. We achieve real-time fusion of the gesture, virtual model, and intelligent equipment in ARCL. Compared with the NOBOOK virtual simulation experiment, ARCL improves the users' real sense of operation and interaction efficiency.

Localization is a fundamental function in cooperative control of micro unmanned aerial vehicles (UAVs), but is easily affected by flip ambiguities because of measurement errors and flying motions. This study proposes a localization method that can avoid the occurrence of flip ambiguities in bounded distance measurement errors and constrained flying motions; to demonstrate its efficacy, the method is implemented on bilateration and trilateration. For bilateration, an improved bi-boundary model based on the unit disk graph model is created to compensate for the shortage of distance constraints, and two boundaries are estimated as the communication range constraint. The characteristic of the intersections of the communication range and distance constraints is studied to present a unique localization criterion which can avoid the occurrence of flip ambiguities. Similarly, for trilateration, another unique localization criterion for avoiding flip ambiguities is proposed according to the characteristic of the intersections of three distance constraints. The theoretical proof shows that these proposed criteria are correct. A localization algorithm is constructed based on these two criteria. The algorithm is validated using simulations for different scenarios and parameters, and the proposed method is shown to provide excellent localization performance in terms of average estimated error. Our code can be found at: https://github.com/QingbeiGuo/AFALA.git.