This study addresses the challenge of multi-robot cooperative exploration under limited local observations in environments with dynamic robot populations. To achieve efficient area coverage within constrained timeframes, we propose the Multi-Robot Informative Planner (MIP), a novel reinforcement learning (RL)-based planning module. The core component of MIP is the Neighborhood Information Aggregator, which employs a graph neural network (GNN) to integrate local neighborhood information for each robot. Our design enhances sample efficiency by minimizing information requirements while ensuring scalability across environments with varying robot numbers. To generate high-quality, expressive neighborhood feature representations, we utilize Graphical Mutual Information (GMI) to maximize the correlation between neighboring robots’ input features and their high-level hidden representations. Furthermore, MIP incorporates the Spatial-Neighborhood Transformer, which captures spatial features and inter-robot interactions through spatial self-attention mechanisms. These components collectively form the Multi-Robot Neural Informative Mapping (MRNIM) framework, outperforming traditional benchmarks in Habitat simulator.

Players-based emotion recognition can help the understanding game players’ emotional states, contributing to the improvement of the game's quality and value. This article develops a hybrid neural network learning framework called convolutional smooth feedback fuzzy network (CSFFN) to detect a player's emotional states in real-time during a gaming process based on electroencephalogram (EEG) signals. Specifically, CSFFN rationally combines a convolutional neural network (CNN), a fuzzy neural network (FNN), and a recurrent neural network (RNN). CNN not only captures spatial characteristics between EEG signals from different channels but also eliminates noise from EEG signals, improving the accuracy and anti-noise performance in game emotion recognition. FNN extracts the membership degree of a player's different emotional states, further improving the emotion recognition accuracy. Since a player's current emotional state is influenced by the previous emotional states during the game process, RNN is employed to capture the temporal characteristics of EEG signals, better improving the emotion recognition accuracy. Experimental results show that CSFFN has higher recognition accuracy and noise resistance in identifying four emotional states (happiness, sadness, superiority, and anger) compared to support vector machine (SVM) with different kernels, linear discrimination analysis (LDA), AlexNet, and VGG16 methods.

Existing driving fatigue detection methods rarely consider how to effectively fuse the advantages of the electroencephalogram (EEG) and electrocardiogram (ECG) signals to enhance detection performance under noise conditions. To address the issues, this article proposes a new type of the deep learning (DL) framework based on EEG and ECG called the product fuzzy convolutional network (PFCN). It should be noted that this article first investigates how to fuse EEG and ECG signals to deal with driving fatigue detection under noise conditions in both simulated and real-field driving environments. Specifically, the PFCN includes three subnetworks. The first uses a fuzzy neural network (FNN) with feedback and a product layer, effectively capturing the particularity and temporal variation of high-dimensional EEG signals and reducing the time-space complexity. The second subnetwork uses a 1-D convolution to convert the ECG data into feature sequences, providing high accuracy and low computational complexity in ECG data classification. The third subnetwork proposes a fusion-separation mechanism to effectively fuse the extracted ECG and EEG features, suppressing the noise interference and ensuring higher detection accuracy. To evaluate the performance of PFCN, a series of experiments has been set up in both simulated and real-field driving environments. The results indicate that the proposed PFCN model has better robustness and detection accuracy compared with several mainstream fatigue detection models.

Beamforming design in non-line-of-sight (N-LOS) multipath channel is challenging for millimeter-wave (mmWave) user equipment (UE), especially when hardware imperfections introduce random phase distortions to the received signals. In this paper, we propose a novel analog beam codebook design algorithm for UE, called approximate correlation matrix (ACM). In the proposed algorithm, beamforming vector is designed according the reference signal received power (RSRP), without the need for exact phase information of received signals. In particular, we exploit the correlation among antennas from phaseless measurements, and derive the precoding/combining vector through a simple Fourier transform. Simulation results show that the proposed algorithms can achieve near-optimal performance with low complexity.

Compressive channel estimation is an effective way to reduce the number of measurements for fast beam tracking in millimeter wave (mmWave) communications. However in some practical scenarios, the phase of the received signal is difficult to measure, leading to challenges for fast beam tracking, especially in non-line-of-sight (NLoS) channels. In this paper, we propose a novel beam tracking algorithm under random phase offset scenarios using compressive sensing (CS). Unlike traditional algorithms based on complex signal measurements, the proposed algorithm could derive the channel information using reference signal received power (RSRP), without the need of knowledge of the phase. To recover the compressive channel with high precision and low complexity, we design a deep learning beam tracking scheme utilizing a complex-valued auto-encoder. Simulation results show that the proposed scheme can outperform traditional hierarchical search manner under blockage and rotation scenarios.

Most existing teleoperation technologies cannot guide robots to complete tasks quickly and accurately in unstructured environments. Therefore, this article proposes an intelligent interaction framework for teleoperation based on human-machine cooperation. The framework is divided into three layers: 1) perception, 2) decision-making, and 3) execution layers. In the perception layer, machines require high-precision environmental information, and human interaction requires rapid feedback on environmental changes. Therefore, a fast three-dimensional reconstruction method combining rough reconstruction and fine reconstruction is proposed. In the decision-making layer, a bare-hand interaction method based on natural interaction and an alignment assistance method based on constraint recognition are proposed. The combination of these two methods realizes the coordinated control of robot movement by humans and computers. In the execution layer, a vision-based error compensation method for assistance is proposed to decrease measurement errors and delays and reduce differences between virtual and real scenes. Finally, experimental results obtained by 15 nonprofessional volunteers show that the proposed method is efficient and user friendly.

Existing visual-based fatigue detection methods usually monitor drivers’ fatigue by capturing their facial features, including eyelid movements, yawn frequency and head pose. However, these approaches typically do not take drivers’ biological signals into consideration. An accurate model for fatigue detection requires combining both facial behavior and biological data. This paper proposes a novel non-intrusive method for driver multimodal fusion fatigue detection by extracting eyelid features and heart rate signals from the RGB video. The multimodal feature fusion method could significantly increase the accuracy of fatigue detection. Specifically, we established two fatigue detection models based on heart rate and the PERCLOS value respectively with one-dimensional Convolutional Neural Network (1D CNN), where the PERCLOS refers to the percentage of eyelid closure over the pupil. Finally, the outputs of the two models are weighted to achieve the multimodal fusion fatigue detection. Simulation results show that our method yield better performance than traditional methods.

Driver fatigue monitoring is very important for driving safety, and many intricate factors while driving make fatigue monitoring harder. To effectively predict driving fatigue, this article proposes a new deep learning framework called TSK-type convolution recurrent fuzzy network (TCRFN) based on the spatial and temporal characteristics of electroencephalogram (EEG) signals. In TCRFN, the convolution block is first introduced to extract spatial dependencies from EEG signals. Furthermore, since EEG noise has a strong spatial dependence, this convolutional neural networks is used to reduce the impact of noise. Additionally, a new local feedback method in fuzzy neural network is proposed to process the EEG signals, which can better capture the temporal dependencies from EEG signals. Finally, a logarithmic spatial firing layer function is used in the proposed TCRFN. The activation performance of this function is smoother, which allows more feature numbers and provides better prediction. The experimental results show that the proposed TCRFN model has better antinoise performance and prediction accuracy compared with other widely used and state-of-the-art models.

Joint angle error of robotic arm has great impacts on the accuracy of the end-effector, which is critical in industrial applications. Therefore, in this article, an online cognitive joint angle error compensation method based on incremental learning is proposed to reduce joint angle error. The proposed method consists of a joint angle error solver and a compensation module, which ensure that the robot can obtain effective joint angle compensation in various situations. The joint angle error solver is used to solve joint angle error online. It uses the redundant constraint method for multilink position measurement so as to calculate the position error of the robot accurately later. The compensation module uses the self-feedback incremental fuzzy neural network (SFIFN) to predict and update the compensation in real time. SFIFN is a variant of the fuzzy neural network (FNN), which uses long short-term memory to introduce a feedback mechanism based on FNN. The incremental learning capability of SFIFN reduces the time for solving error and makes the module runs in real time. Specifically, two inertial measurement units mounted at the ends of links are used to measure pose changes of the ends of corresponding links. Both the simulated and the real experiments show that the proposed method yields good compensations to joint angle error and its potentials for smart manipulation.

Driving fatigue can be detected by measuring drivers' heart rate with a wearable device or extracting their facial features with an RGB camera. However, a wearable device causes inconvenience and discomfort to the driver, and an RGB camera's detection accuracy may be affected by light, glasses, and head orientation. Furthermore, most existing methods ignored the temporal information of fatigue features and the relationship between the features, lowering recognition accuracy. Additionally, some existing fatigue detection methods focused on dealing with fatigue features with a temporal slice, ignoring temporal variations in the features. To address these problems, a single RGB-D camera is first used to extract three fatigue features: heart rate, eye openness level, and mouth openness level. More importantly, this paper proposes a novel multimodal fusion recurrent neural network (MFRNN), integrating the three features to improve the accuracy of driver fatigue detection. Specifically, a recurrent neural network (RNN) layer is applied in the MFRNN to obtain the temporal information of the features. Since the heart rate feature is a physiological signal extracted indirectly, it contains more noise and is fuzzier than the other features. To deal with the fuzziness and noise, we combine fuzzy reasoning with RNN to extract the temporal information of the heart rate. To identify the relationship between the features, we develop a new relationship layer containing a two-level RNN, for which the input is the temporal information of the features. Both the simulation and field experiment results show that the proposed method provides better performance than similar methods.

Real-time recognition of driver emotions can greatly improve traffic safety. With the rapid development of communication technology, it becomes possible to process large amounts of video data and identify the driver's emotions in real time. To effectively recognize driver's emotions, this paper proposes a new deep learning framework called Convolution Bidirectional Long Short-term Memory Neural Network (CBLNN). This method predicts the driver's emotion based on the geometric features extracted from facial skin information and the heart rate extracted from changes in RGB components. The facial geometry features obtained by using Convolutional Neural Network (CNN) are intermediate variables for the heart rate analysis of Bidirectional Long Short Term Memory (Bi-LSTM). Subsequently, the output of Bi-LSTM is used as input to the CNN module to extract the hear rate features. CBLNN uses Multi-modal factorized bilinear pooling (MFB) to fuse the extracted information and classifies it into five common emotions: happiness, anger, sadness, fear and neutrality. Our emotion recognition method was tested, proving that it can be used to quickly and steadily recognize emotions in real time.

Traditional human-robot interfaces usually have limitations in accuracy and/or operational space. This article proposes a natural human-robot interface using an adaptive tracking method, which can effectively expand the operational space while ensuring high accuracy. The natural interface allows the robot to directly reproduce the user's hand movement, making the interaction more intuitive and natural. The leap motion is fixed on the Cartesian platform to capture the movement of the user's hand. Because the Cartesian platform follows the hand and keeps the hand in the center of the detection area, the measurement accuracy is improved and the measurement space can be extended. During the process of acquiring gesture data, the measurement errors were found to increase over time because of the inherent noise of the sensor. To deal with this problem, the unscented Kalman filter is applied to estimate the position of the hand. Moreover, an adaptive velocity control method is proposed to improve the operation accuracy and reduce the task execution time with the consideration of users' habits and easiness of usage. The effectiveness of this interface is verified by a series of experiments, and the results show that the proposed interface can be used by nonprofessional users for object operation tasks and can provide users with superior interactive experiences.

In this article, a novel human-machine interface, in which two Leap Motion (LM) controllers and a coil are attached to a Cartesian platform to provide contactless electromagnetic force feedback for enhancing the accuracy and efficiency of human-robot manipulation tasks is presented. To implement such an interface, an interval Kalman filter, an improved particle filter, and a mean filter are integrated to estimate accurately the position and orientation of the hand gesture tracked by the two LM controllers, and to smoothen the movement of the Cartesian platform. The back propagation neural network is employed to regulate the electric currents of the coil attached to the Cartesian platform for accurate force feedback. A series of comparative experiments are performed, and the results show that the presented interface greatly improved the efficiency and accuracy of human-robot manipulation tasks in comparison with existing methods, indicating its great potentials for many industry scenarios.

The aim of this paper is to propose a novel markerless human-robot interface, which is derived from the idea that the manipulator copies the movements of human hands. With this method, one operator could control dual robots through both his or her hands in a contactless and markerless environment. In order to obtain the position and orientation of human hands in real time, a sensor called leap motion (LM) is employed in this paper. However, because of the tracking errors and noises of the sensor, the measurement errors increase with time. Therefore, interval Kalman filter (IKF) and improved particle filter (IPF) are used to estimate the position and the orientation of the human hands, respectively. Furthermore, in order to avoid the perceptive limitations and the motor limitations, which prevent the operator from carrying out the high-precision experiment, a modification of adaptive multispace transformation (AMT) method is raised to assist the operator to determine the posture of the manipulator. The greatest strength of our method is that it is totally contactless and could estimate the pose of the human hands accurately and stably without any assistance from markers. A series of experiments have been conducted to verify the human-manipulator interface system, and the results show that the system is indeed of high availability and fault tolerance in teleoperation, which means even a novice can easily and successfully control robots with this human-manipulator interface.

This study explores the feasibility of linear interference alignment (IA) for a ( M × M , 1) K system with multi‐frequency independent channels. The authors prove that number of users K ≤ 2 M − 2 is a necessary condition for ( M × M , 1) K multi‐frequency IA system, which is less than that of ( M × M , 1) K multiple‐input–multiple‐output system (i.e. K ≤ 2 M − 1). The conclusion can be extended to all cases with the diagonal channel matrix. They also reveal that the traditional frequency division multiplexing method is inefficient in spectrum utilisation. They apply max‐SINR (signal to interference plus noise ratio) and iterative IA algorithms and the author's simulation results verify that IA algorithms work well in ( M × M , 1) 2 M –2 IA system.

In this article, we suggest an efficient beam switching technique for the emerging 60GHz wireless personal area networks. Given the pre-specified beam codebooks, the beam switching process, aiming to identify the best beam-pair for data transmissions, is formulated as a global optimization problem in a two-dimension plane that is formed by the potential beam pattern index. As the analytical gradient information of the objective reward function is practically unavailable, Rosenbrock numerical algorithm is properly adopted to implement beam searching, by implicitly approaching and exploiting the gradient descent direction through the numerical pattern-search mechanic. In order to enhance search performance, furthermore, a novel initialization process is presented to provide the feasible initial solution for Rosenbrock search. Inspired by the appealing conception of small-region dividing and conquering, this pre-search algorithm can efficiently reduce the search scope and hence improve the success probability. The developed beam switching technique, i.e. an initialization process followed by Rosenbrock search, exhibits a much lower complexity than the current state-of-the-art strategies. It is demonstrated from both theoretical analysis and numerical experiments that, compared with the existing popular methods, the required protocol overhead of the new beam-training procedure can be significantly reduced, accompanying the power consumption of 60GHz devices.