With the aging of society, assistive companion robots have become an important solution to address the daily needs of the elderly. However, the elderly may suffer from slow thinking and memory loss, which makes it difficult for robots to accurately capture executable intentions during human-robot interaction. To address this challenge, this paper proposes an Adaptive Multimodal Complementary Intent Understanding Approach (AMCIU). First, a self-attentive mechanism is utilized to extract multimodal features of gestures, speech and images. Second, knowledge graph is utilized for the first time to obtain complementary information between image and audio modalities. Finally, robust intent integration is achieved through a multimodal hybrid expert intent fusion technique. In comparison with multiple state-of-the-art multimodal learning methods, AMCIU shows significant performance improvement in the building block tower scenario, where the intention understanding accuracy reaches 96.11%. This paper provides a novel research idea for natural human-robot interaction.

Middle school chemistry labs face challenges like safety risks, high costs, and reagent pollution. Many schools now use intelligent simulation systems, but these often suffer from limited input modes, weak intent understanding, and poor collaboration. This paper proposes an intent recognition algorithm based on dynamic decoding of Hidden Conditional Random Fields (HCRF), integrating voice, gesture, and visual data via LSTM networks. The method improves intent accuracy and interaction fluidity by uncovering hidden inter-modal relationships. An intelligent navigation system also guides users and corrects deviations in procedures. Tests show a 94.29% accuracy rate and positive user feedback.

As the aging population issue worsens, people are experiencing increasing life pressures, and the elderly are unable to receive sufficient care, intensifying the demand for home-based intelligent massage healthcare robots. Addressing the challenges faced by users in expressing preferences for specific areas like the back through gestures or voice during the use of massage robots, and the issue of elderly individuals expressing imprecisely, which hinders effective communication of the desired massage target location to the robot, this paper proposes a new human-machine interaction mode for massage robots. The main innovations include: achieving precise control of the massage area through the representation of a virtual human body on the screen; proposing a virtual-to-real mapping algorithm based on human acupoints to address the consistency issue in human virtual-to-real mapping; introducing a multimodal intent understanding algorithm based on dynamic information entropy to address the current limitations of a single interaction mode and low intent comprehension rates in massage robot interactions. Experimental results demonstrate that the proposed multimodal massage localization algorithm achieves good massage intent recognition effects. By combining natural human-machine interaction with intent understanding, it not only accurately captures user massage intentions and assists in completing massage tasks but also reduces user psychological and cognitive loads, leading to more desirable interaction outcomes.

The current teaching mode in geometry mainly focuses on two-dimensional levels, and the teaching tools utilized are static and not interactive. This paper proposes the use of a smart pen for three-dimensional geometry experimental teaching and a multimodal intention understanding and human-computer collaboration algorithm for the smart pen. The primary innovations of this paper lie in the development of a smart pen and a virtual platform for geometry education tailored for geometry experimental instruction. This system can promptly perceive and comprehend user behavior in real-time. Furthermore, a standardized topological equivalence model is proposed as the basis for a point selection strategy. By establishing correspondence between the modeled point selection model and the actual operation scene, the behavioral intent imposed on the model is applied to the operation object of the actual scene. Additionally, CNN-based and information entropy-based multimodal fusion intention understanding models are proposed for different input modalities to capture the operational intention of users by fusing their multimodal input data. The algorithm further improves the accuracy rate through an error correction mechanism based on implicit interaction to achieve better human-computer collaboration. The algorithm proposed in this paper has resulted in a 0.47-second improvement in point selection average time and achieved an intent understanding accuracy of 98.21%. This improvement leads to better fault tolerance and fluency during human-computer interaction, reduces the cognitive load on the user, and improves the overall user experience.

During the COVID-19 pandemic, online classes became the only option for many students. The main challenge for these classes was conducting risky and complex chemical or biological experiments in a domestic environment. To address this challenge, a smart experiment system called MRLab was developed. MRLab used wearables such as a smart glove and head-mounted device to record sensory data and a multimodal hybrid fusion model GVVS to interpret the user’s experimental intent, which essentially transforms the user’s abstract behavioral actions into a probabilistic set of experimental intent that can be computed. Different experiments in MRLab used different libraries of experimental intents. The SrNet model in GVVS was used to estimate the probability of the user’s gesture behavior generated from the smart glove, while the SIPA algorithm compared speech information entered during the experiment with the experimental intent library to estimate the probability of the user’s intent. At the same time, the scene visual channel monitored the information about the object the user intended to operate, with the SVF algorithm computing the probability of the intended object in real-time. The results from ANOVA and post-hoc comparative testing conducted on 21 volunteers revealed that MRLab outperformed other experiment modes, including WEB, AR, and VR, with a higher intention understanding rate, efficiency, and user satisfaction. Therefore, MRLab proved to be a useful alternative to traditional physics laboratory experiments during the pandemic, along with being an additional teaching tool for remote learning purposes.

The naturalness and safety of human-computer interaction have always been primary research focuses in the field of human-computer interaction. This paper proposes a multimodal intention understanding algorithm (MIUIC), which incorporates comfort analysis, as a solution to address the issues of low intention understanding rate, weak interaction, and weak collaboration that are often observed in most massage systems. The algorithm efficiently fuses multimodal data based on objective implicit information to address the challenge of low intention understanding rates caused by non-standard expression of natural behavior. Moreover, this algorithm incorporates comfort reasoning to detect and address intentions related to security threats while providing the ability for robots to make behavioral decisions through inverse active interaction, leading to more equitable human-robot interactions. To test the validity and safety of the MIUIC algorithm, we embedded the algorithm into a mechanical arm massage system. Subsequently, 45 elderly volunteers were invited to participate in experimental tests. Finally, to verify the validity and safety of the MIUIC algorithm, we assessed the algorithm in terms of four aspects, including multimodal intention recognition rate, the ability to reduce data dispersion, the intention enhancement rate under reverse human-machine interaction, and the rate of avoiding dangerous intentions. In conclusion, the MIUIC algorithm enhances the intention understanding rate and promotes.

Abstract Various deep learning‐based video frame interpolation methods have been proposed in the past few years, but how to generate high quality interpolated frames in videos with large motions, complex backgrounds and rich textures is still a challenging issue. To deal with this limitation, a frame interpolation method based on residual blocks and feature pyramids is proposed. U‐Net is the main architecture of our method, which can capture multi‐layer information, segment objects from the background and obtain parameters with motion information to guide frame interpolation. However, the upsampling and subsampled of U‐Net will lose important information. In order to acquire more detailed contextual information, shortcut connection is used in the encoder basic module. At the same time, feature pyramid network is employed to capture features at different scales of the decoder to improve the representation of inter‐frame spatial‐temporal features. The experimental results show that the proposed method outperform the baseline methods in both of objective and subjective evaluations on different datasets. In particular, the method has obvious advantages on datasets which contain complex background.

Robot grasping encompasses a wide range of research areas; however, most studies have been focused on the grasping of only stationary objects in a scene; only a few studies on how to grasp objects from a user's hand have been conducted. In this paper, a robot grasping algorithm based on deep reinforcement learning (RGRL) is proposed. The RGRL takes the relative positions of the robot and the object in a user's hand as input and outputs the best action of the robot in the current state. Thus, the proposed algorithm realizes the functions of autonomous path planning and grasping objects safely from the hands of users. A new method for improving the safety of human–robot cooperation is explored. To solve the problems of a low utilization rate and slow convergence of reinforcement learning algorithms, the RGRL is first trained in a simulation scene, and then, the model parameters are applied to a real scene. To reduce the difference between the simulated and real scenes, domain randomization is applied to randomly change the positions and angles of objects in the simulated scenes at regular intervals, thereby improving the diversity of the training samples and robustness of the algorithm. The RGRL's effectiveness and accuracy are verified by evaluating it on both simulated and real scenes, and the results show that the RGRL can achieve an accuracy of more than 80% in both cases. RGRL is a robot grasping algorithm that employs domain randomization and deep reinforcement learning for effective grasping in simulated and real scenes. However, it lacks flexibility in adapting to different grasping poses, prompting future research in achieving safe grasping for diverse user postures.

At present, the teaching of experiments in primary and secondary schools is affected by cost and security factors. The existing research on virtual-experiment platforms alleviates this problem. However, the lack of real experimental equipment and the use of a single channel to understand users’ intentions weaken these platforms operationally and degrade the naturalness of interactions. To slove the above problems,we propose an intelligent experimental container structure and a situational awareness algorithm,both of which are verified and then applied to a chemical experiment involving virtual-real fusion. First, acquired images are denoised in the visual channel, using maximum diffuse reflection chroma to remove overexposures. Second, container situational awareness is realized by segmenting the image liquid level and establishing a relation-fitting model. Then, strategies for constructing complete behaviors and making priority comparisons among behaviors are adopted for information complementarity and information independence, respectively. A multichannel intentional understanding model and an interactive paradigm fusing vision, hearing and touch are proposed. The results show that the designed experimental container and algorithm in a virtual chemical experiment platform can achieve a natural level of human-computer interaction, enhance the user's sense of operation, and achieve high user satisfaction.

EHRs contain the patient’s historical disease information, and a natural idea is to predict the patient’s disease or diagnose the patient based on EHRs. However, existing deep learning models using EHRs are not satisfactory in solving several key challenges: 1) most of the existing methods are lack of priori knowledge assistance during model learning; and 2) higher-order relationships among diseases and patients are not explored sufficiently. To address these issues, in this paper we propose a hypergraph-based deep learning model for disease prediction, namely EHR2HG. First, we propose to utilize the existing disease classification information to give the model a better initialization condition. Second, by constructing hypergraphs, we consider all of the patients together instead of separate individuals, so our models can model entity relationships such as comorbidities or patient class groups. We have evaluated the proposed model on a real-world EHRs dataset and the results demonstrated that EHR2HG can achieve comparable and better performance than several state-of-the-art baseline methods.

In this work, we propose an efficient retrieval method for human motion capture (MoCap) data based on supervised deep hash code learning. Raw Mocap data is represented into three 2D images, which encode the trajectories, velocities and self-similarity of joints respectively. Such image-based representations are fed into a convolutional neural network (CNN) adapted from the pre-trained VGG16 network. Further, we add a hash layer to fine-tune the CNN and generate the hash codes. By minimizing the loss defined by classification error and constraints on hash codes, highly discriminative hash representations of the motion data can be generated. As experimentally demonstrated on the public HDM05 data set, our algorithm achieves high accuracy comparing with the state-of-the-art MoCap data retrieval algorithms. Besides, it achieves high efficiency due to the fast matching of hash codes.

The aging population phenomenon is increasing daily, making the realization of robots to help the elderly via human-robot collaboration an urgent need. However, in terms of the current research, there are many challenges in the realization of collaborative systems applied to help the elderly. The interaction methods used in related studies are single and unfavorable for the elderly, which not only affects the understanding of users' intentions but also brings burdens to the elderly. Moreover, these approaches generally ignore the flexibility of the division of labor and the degree of human need for assistance, which reduces the performance of goal completion and affects the elderly's evaluation of the robot escort. To this end, this paper designs and implements a human-robot collaboration system for helping the elderly (HRCS_EE). A multimodal fusion algorithm based on understanding and supplemented by inquiry is proposed. Additionally, a collaborative division strategy based on a cognitive model is researched, and a decision-making algorithm that aims to achieve the most efficient completion of collaborative tasks and minimize the burden on people is proposed. Finally, the method is verified and applied in a human-robot collaboration system based on repeated interaction. The experimental results show that the system can accurately and reliably obtain users' intentions in complex environments and correctly make division of labor decisions in line with the environment, so that the robot can provide the assistance that is truly needed by the elderly, which is well received by users.

At present, most experimental teaching systems lack guidance of an operator, and thus users often do not know what to do during an experiment. The user load is therefore increased, and the learning efficiency of the students is decreased. To solve the problem of insufficient system interactivity and guidance, an experimental navigation system based on multi-mode fusion is proposed in this paper. The system first obtains user information by sensing the hardware devices, intelligently perceives the user intention and progress of the experiment according to the information acquired, and finally carries out a multi-modal intelligent navigation process for users. As an innovative aspect of this study, an intelligent multi-mode navigation system is used to guide users in conducting experiments, thereby reducing the user load and enabling the users to effectively complete their experiments. The results prove that this system can guide users in completing their experiments, and can effectively reduce the user load during the interaction process and improve the efficiency.

Augmented reality classrooms have become an interesting research topic in the field of education, but there are some limitations. Firstly, most researchers use cards to operate experiments, and a large number of cards cause difficulty and inconvenience for users. Secondly, most users conduct experiments only in the visual modal, and such single-modal interaction greatly reduces the users' real sense of interaction. In order to solve these problems, we propose the Multimodal Interaction Algorithm based on Augmented Reality (ARGEV), which is based on visual and tactile feedback in Augmented Reality. In addition, we design a Virtual and Real Fusion Interactive Tool Suite (VRFITS) with gesture recognition and intelligent equipment. The ARGVE method fuses gesture, intelligent equipment, and virtual models. We use a gesture recognition model trained by a convolutional neural network to recognize the gestures in AR, and to trigger a vibration feedback after a recognizing a fivefinger grasp gesture. We establish a coordinate mapping relationship between real hands and the virtual model to achieve the fusion of gestures and the virtual model. The average accuracy rate of gesture recognition was 99.04%. We verify and apply VRFITS in the Augmented Reality Chemistry Lab (ARCL), and the overall operation load of ARCL is thus reduced by 29.42%, in comparison to traditional simulation virtual experiments. We achieve real-time fusion of the gesture, virtual model, and intelligent equipment in ARCL. Compared with the NOBOOK virtual simulation experiment, ARCL improves the users' real sense of operation and interaction efficiency.

The microstructure images of cement are often used as the main data source for estimating compressive strength. They contain ample physical properties during the hydration process. Different gray values represent different substances in the grayscale image of cement. Deep learning algorithm based on microstructure images have been proposed to estimate cement compressive strength (CCS). However, there are a large number of parameters that need to be adjusted in deep structure. The high-efficiency system named broad learning system (BLS) is tried to use to estimate the cement compressive strength. The original cement microstructure images and the extracted features are used as input respectively, the connection weights can be obtained directly by calculating pseudo inverse matrix of feature matrix of microstructure image. If the structure is not sufficient to gain suitable result, BLS only calculates the pseudo inverse matrix of additional nodes to improve accuracy. The experiment shows that the broad learning structure (BLS) is an effective and efficient method on estimating cement compressive strength by contrasting with deep learning structure.

Point set registration is a fundamental problem in many domains of computer vision. In previous work on the registration, the point sets are often represented using Gaussian mixture models and the registration process is represented as a form of a probabilistic solution. For non‐rigid point set registration, however, the asymmetric Gaussian (AG) model can capture spatially asymmetric distributions compared with symmetric Gaussian, and the structural feature of the point sets reserve relatively complete and has important significance in registration. In this work, the authors designed a new shape context (SC) descriptor which combines the local and global structures of the point set. Meanwhile, they proposed a non‐rigid point set registration algorithm which formulates a registration process as the mixture probability density estimation of the AG mixture model, and the method introduce the structural feature by the new SC. Extensive experiments show that the proposed algorithm has a clear improvement over the state‐of‐the‐art methods.

A motion-compensated frame interpolation (MCFI) algorithm for multiview video based on inter-view and intra-view correlations is proposed in this paper. First, unidirectional motion estimation (ME) is implemented to obtain forward and backward motion vector fields (MVFs). Subsequently, occlusion blocks in previous and current frames are detected. Then, an inter-view-correlation based method is adopted for occlusion handling. After that, the motion vector (MV) outliers are detected and corrected by considering spatial, color and depth information of current viewpoint video. Finally, MVs are assigned to the interpolated frame for frame reconstruction. Experimental results demonstrate that the proposed algorithm provides better performance than existing 2D and 3D video MCFI methods.

The implementation of user's interactive goal is completely dependent on the recognition rate of gesture in smart TV's interaction. This increases the user's interactive load. So we propose an implicit interaction algorithm for smart TV based on interaction context. Firstly, we implement the data's fusion and feature's extraction of interactive context. Then, we designed an implicit interaction model(MLC-FCM) which is based on interaction context. Finally, MLC-FCM is used to realize the perception of the user's intention to complete the interaction task. This algorithm can improve the accuracy of operations, reduces the time spending and gesture's movement distance.

On the basis of research on gesture recognition algorithm to the static gesture as a breakthrough idea of using static gesture recognition to identify the dynamic gestures. Firstly, this paper used Kinect to obtain depth image and RGB color image segmenting a hand. After this , for each frame this paper used the skin color segmentation method to process the image. After extracting the hand images, we used the method of density distribution feature and recognition of fingers' numbers based on the binary images to extract the hand gesture feature, then with pre-established gesture library template matching, the final choice of the closest gesture is the recognition of hand gestures. Experimental results demonstrate the accuracy of the proposed algorithm is higher, and successfully applied to the man-machine finger-guessing game,which made a good user experience.

This paper proposes a novel motion compensated frame interpolation (MCFI) method based on adaptive hybrid motion estimation and compensation (AHMEC) for 3D video. In our work, we deal with the problem of ghost artifacts around the foreground object boundaries by considering motion and depth information jointly. First, the motion vector field (MVF) of the interpolated frame is estimated using block-based method. We use depth and motion information to distinguish the occlusion areas in the interpolated frame. Then, an adaptive pixel-based motion estimation (ME) method is applied to detail the MVF in the covering and uncovering areas. Simulation results show that the proposed MCFI algorithm outperforms the conventional algorithms in terms of objective and subjective performances.

A frame rate up-conversion (FRUC) method for 3D video (3DV) is presented in this paper. Inspired by the fact that moving foreground objects draw more attention of the viewers, in our method depth guided extended block matching (DGE-BM) is adopted to maintain the completeness of the foreground object. We first obtain the motion vector field (MVF) of the interpolated frame via block-based bi-directional motion estimation (ME). And the blocks of the interpolated frame are classified according to the depth information. Then, the boundary blocks are divided into sub-blocks, whose motion vectors (MVs) are estimated using DGE-BM. Finally, the refined MVF is applied to do motion compensation. Experimental results show that the frame interpolation quality of the proposed method achieves significantly improvement comparing with existing algorithms.

ABSTRACT In this paper, we propose a novel genetic algorithm approach to human motion capture (MoCap) data segmentation. For a given MoCap sequence, it constructs a symbolic representation through unsupervised sparse learning, detects the candidate segmenting points to the sequence, models the selection/deselection of each candidate with a gene, and employs the genetic algorithm to find the optimal solution. To the best of our knowledge, we for the first time introduce the genetic algorithm and the sparse learning technique to the problem of MoCap data segmentation, leading to excellent segmentation performance as experimentally demonstrated. Copyright © 2014 John Wiley & Sons, Ltd.