Multimodal peripheral physiological signal fusion for emotion recognition seeks to perceive or recognize human emotions using peripheral modalities such as electromyography, electrodermal activity, and respiratory wave. Previous approaches to multimodal fusion primarily focus on emotion-sensitive signals such as electroencephalogram (EEG), often overlooking the potential value of peripheral physiological signals in emotion recognition. Moreover, the inherent heterogeneity among different modalities continues to pose challenges to fusion quality. In this article, we propose a multimodal decoupled multimodal fusion (DMF) to address these issues, enabling flexible feature decoupling, cross-modal feature interaction, and relational knowledge learning. Specifically, each modality’s representation is first decoupled into two components: common features and modality-specific features; second, the DMF employs progressive cross attention to facilitate the exchange of modality-specific features across different modalities; and finally, it uses relational knowledge to learn multimodal spliced features, embedding both inter-modal and intra-modal feature relationships. DMF offers a dynamic multimodal emotion recognition framework that leverages the emotional information contained in diverse modalities. Experimental results demonstrate that the DMF method consistently outperforms previous approaches and provides a viable solution for multimodal peripheral physiological signal fusion.

Existing approaches to multi-person pose tracking often suffer from low-confidence detections due to inter-instance and intra-instance occlusions, as well as non-canonical poses. In this work, we propose a novel solution by addressing two critical aspects: incomplete joint temporal dependencies and spatio-temporal voxelization. First, we introduce a method for extracting hierarchical relationships between joints based on human dynamics, enabling the model to reason about occlusions within the spatial topology of the human body. This hierarchical approach tackles incomplete joint visibility by leveraging the interdependencies between joints in both space and time. Second, we present a spatio-temporal occupancy network for multi-person pose tracking. By stacking 2D pose data over time to create a spatio-temporal voxel grid, the model captures temporal relationships between instances and joints, enhancing spatio-temporal correlations and learning keypoint distributions under occlusions or non-canonical poses. Extensive experiments on the PoseTrack2017, PoseTrack2018, and PoseTrack21 dataset demonstrate that our method improves multi-person pose tracking performance, achieving state-of-the-art mAP.

The combination of wearable sensors and competitive sports provides quantitative information for scientific training, effectively assisting athletes in improving their athletic performance. This study presents a technical framework for athletic sports assessment in competitive swimming based on body-area sensor networks. In our approach, wearable inertial sensor nodes are placed on specific body parts of the athletes to capture motion data during different competitive swimming strokes. Multiwearable inertial sensor nodes are worn on specific body parts of athletes for real-time monitoring motion data during training sessions. A motion intensity detection-based error-state-Kalman-filter algorithm is proposed for multisensor data fusion. Additionally, through kinematic statistical analysis, the characteristics of joint motion during training are clearly explained. Furthermore, a deep learning network that fuses sensor time series and human skeleton graphs is proposed for different stroke phase segmentation, enabling quantitative measurement of motion phases, and several baseline classifiers are chosen for comparison to validate the robustness of our phase segmentation method. We also investigate the sensor combination selection issue during the phase segmentation process to determine the optimal sensor configuration. Our approach provides a scientific solution for the integration of wearable sensors and competitive sports, contributing to the high-quality development of the next generation of smart sports.

Domain generalization remains a key challenge in human activity recognition based on channel state information (CSI). Different domains correspond to distinct data distributions, deviating from the typical assumption of independent and identically distributed (i.i.d.) data, which leads to significant performance degradation when models are applied to unseen domains. To address this issue, we propose a novel domain generalization model that integrates meta-learning initialization and an adaptive channel grouping attention mechanism. First, a meta-learning strategy is employed to acquire well-initialized parameters from multiple source domain tasks, enabling the model to implicitly enhance its cross-domain generalization ability. Second, an adaptive grouping attention mechanism is designed in the feature extraction stage to effectively capture the sensitivity differences of different subcarriers to human activities. Meanwhile, a random masking training mechanism is introduced to simulate real-world domain variations and improve model robustness. In addition, a domain adversarial training framework based on the gradient reversal layer (GRL) is adopted to mitigate domain-specific feature dependency, further enhancing the model's generalization capability. We evaluate our proposed method on both a self-collected dataset, which includes human activity data from nine volunteers across six different environments, and a public CSI dataset. The experimental results demonstrate that our method significantly outperforms existing approaches in domain generalization performance, verifying its effectiveness and practical applicability.

Car detection is an important task that serves as a crucial prerequisite for many automated driving functions. The large variations in lighting/weather conditions and vehicle densities of the scenes pose significant challenges to existing car detection algorithms to meet the highly accurate perception demand for safety, due to the unstable/limited color information, which impedes the extraction of meaningful/discriminative features of cars. In this work, we present a novel learning-based car detection method that leverages trichromatic linear polarization as an additional cue to disambiguate such challenging cases. A key observation is that polarization, characteristic of the light wave, can robustly describe intrinsic physical properties of the scene objects in various imaging conditions and is strongly linked to the nature of materials for cars (e.g., metal and glass) and their surrounding environment (e.g., soil and trees), thereby providing reliable and discriminative features for robust car detection in challenging scenes. To exploit polarization cues, we first construct a pixel-aligned RGB-Polarization car detection dataset, which we subsequently employ to train a novel multimodal fusion network. Our car detection network dynamically integrates RGB and polarization features in a request-and-complement manner and can explore the intrinsic material properties of cars across all learning samples. We extensively validate our method and demonstrate that it outperforms state-of-the-art detection methods. Experimental results show that polarization is a powerful cue for car detection. Our code is available at https://github.com/wind1117/AAAI24-PCDNet.

In a closed environment lacking global positioning system (GPS) signals, how to achieve accurate navigation and positioning is a very challenging task. Zero velocity update (ZUPT) is a highly effective foot-mounted inertial pedestrian navigation systems in such environment. However, despite its effectiveness, the limitation of accurate detecting the zero-velocity-interval (ZVI) and heading drift are still the significant challenges of the ZUPT method. To address these issues, a deep learning method for adaptive ZVIs detection is established based solely on inertial sensors by comparing with the optical motion capture system. Additionally, an improved ZUPT-aided extend Kalman filter (EKF) divides the measurement updates of the ZVIs is established for multisensor data fusion, and the heading change with heuristic drift reduction (HDR) is also adopt as measurement, thereby yielding to limit the heading drift. Experimental results demonstrate that our method provides a better estimate of the heading angle, as well as more accurate ZVIs detection, leading to more precise dead-reckoning position estimates than other state-of-the-art methods.

The classification analysis of incomplete and imbalanced data is still a challenging task since these issues could negatively impact the training of classifiers, which were also found in our study on the physical fitness assessments of patients. And in fields such as healthcare, there are higher requirements for the accuracy of the generated imputation values. To train a high-performance classifier and pursue high accuracy, we attempted to resolve any potential negative impact by using a novel algorithmic approach based on the combination of multivariate imputation by chained equations and the ensemble learning method (MICEEN), which can solve the two problems simultaneously. We used multivariate imputation by chained equations to generate more accurate imputation values for the training set passed to ensemble learning to build a predictor. On the other hand, missing values were introduced into minority classes and used them to generate new samples belonging to the minority classes in order to balance the distribution of classes. On real-world datasets, we perform extensive experiments to assess our method and compare it to other state-of-the-art approaches. The advantages of the proposed method are demonstrated by experimental results for the benchmark datasets and self-collected datasets of physical fitness assessment of tumor patients with varying missing rates.

The lower extremity exoskeleton, which can sense the neural motion state of the human body and then provide motion assistance, is gradually replacing the traditional wheelchairs and assistive devices, making many patients with disabilities or movement disorders able to regain the walking function. This survey provides a comprehensive review on recent technological advances in lower extremity neurorehabilitation exoskeleton from the perspectives of sensing, gait dynamics, and human–robot collaboration. For each technology category, a detailed comparison among state-of-the-art solutions is provided. The results show that the exoskeleton has been greatly improved in mechanical and learning ability. However, some issues, such as adaptability, safety, and efficiency still restrict the development of exoskeleton technology. To address these problems, the remaining open challenges and future directions to improve intelligence, sensing, gait analysis, trust, efficiency, generalization, and power consumption of exoskeleton are also presented and discussed.

Tremors are typically experienced by patients at the beginning of Parkinson's disease (PD). Clinicians evaluate clinical symptoms based on scale and experience, but mild tremors do not have significant characteristics and are difficult to observe with the naked eye. Implementing an intelligent and objective method to identify PD patients with early tremor symptoms and healthy controls (HC) is necessary. This study used wearable sensors to collect 9-axis inertial signals and 2-channel sEMG signals at the wrists of 13 HC and 24 PD patients from Dalian Municipal Central Hospital. Based on a Long Short-Term Memory Network (LSTM), an attention mechanism, and a Fully Convolutional Network (FCN), we develop a model to classify and recognize data from PD patients. Compared the proposed method with several classification methods, the results showed that the proposed method achieved higher classification accuracy (91.78%), precision (100%), recall (87.50%), and F1-score (93.33%) of PD class than Support Vector Machine, FCN, and LSTM. The computing time of the proposed method is approximately 1 second. The proposed method identifies early PD patients by pre-clinical assessment of mild upper limb tremors, which is valuable for early treatment and rehabilitation.

To address the law enforcement challenges arising from the rapid expansion of the online catering industry, China’s market regulatory authorities are pursuing countermeasures through legislative efforts and innovative regulatory models. At present, administrative law enforcement for online catering faces difficulties in license management and ensuring the authenticity and security of data. Specifically, there is a prevalence of fraudulent licenses in the industry, and the market supervision department struggles to verify the authenticity of the data gathered during investigations and evidence collection. The tamper-proof and transparent features of blockchain technology cannot be directly applied to the online catering domain. Consequently, to attain trusted license management and secure data-sharing measures, further system design is necessary. To achieve trusted license management, we employ blockchain technology for managing qualification certificates and delineate the format for valid certificates and the application process. The secure data-sharing protocol is divided into two stages. In the first stage, we ensure data authenticity before uploading it to the blockchain through a consensus process involving platforms, users, and merchants. In the second stage, we implement fine-grained access control based on the on-chain data from the previous stage, utilizing conditional proxy reencryption and ultimately employing the data digest on the blockchain for verification. The experiment thoroughly validates the system’s functionality, which is traceable, transparent, and scalable. Moreover, the minimal delay introduced by ensuring data authenticity is virtually negligible. The system’s additional storage overhead does not surpass 10%, remaining within acceptable limits.

Gait analysis is a prosperous tool for the clinical evaluation and diagnosis. In this article, a portable gait analysis system based on foot-mounted inertial sensors is established. A threshold-free method using a long short-term memory recurrent neural network is constructed to segment four typical gait phases in a gait sequence for the temporal parameters analysis. Segmentation accuracy reaches over 95% across recruited subjects with distinct gait patterns, which is significantly superior when compared with traditional machine learning methods. The zero-velocity indicator is generated successively according to the segmented sequence to accomplish zero velocity update for the spatial parameter calculation. The accuracy of the proposed system is also validated through the OptiTrack in the lab. The comparison result of the stride length shows that the error between the two systems is less than 2%, which demonstrates that our system can satisfy the demand in the clinical.

Using inertial measurement units mounted on foot is a feasible approach to improve the positioning accuracy for the human motion capture system. This paper presents a lightweight and low cost wireless inertial motion capture system for the simultaneous reconstruction of human body attitude and displacement. First of all, the device is based on human sensor networks and distributes 15 sensor nodes on the key human limbs. Then, after an initial sensor alignment with the reduced error, a zero-speed update algorithm is used to calculate foot displacement. In addition, to constantly update the human posture information, a kind of motion reconstruction method based on the gradient descent method was used to fuse the sensor data. Finally, a new method of three-dimensional human body reconstruction is proposed, which is different from the traditional motion capture system. Through unconstrained traversal of the root, the human posture and foot trajectory are combined to realize the synchronous reconstruction of posture and displacement. It is concluded from the experiment results that the estimation errors are well controlled, and motion patterns are consistent with the actual situation.

Wearable inertial motion capture, a new type of motion capture technology, mainly estimates the human posture in 3-D space through multisensor data fusion. The available method for sensor fusion is usually aided by magnetometers to remove the drift error in yaw angle estimation, which in turn limits their application in the presence of a complex magnetic field environment. In this article, an extended Kalman filter (EKF) data fusion method is proposed to fuse the 9-axis sensor data. Meanwhile, the heuristic drift reduction (HDR) method is used to calibrate the accumulated error of a heading angle. In addition, the position in 3-D space is estimated by the foot-mounted zero-velocity-update (ZUPT) technique. Combining 3-D attitude and position, a biomechanical model of the human body is established to track the motion of a real human body. The EKF algorithm and position estimation methods are benchmarked against the golden standard, optical motion capture system, for various indoor experiments. In addition, various outdoor experiments are also conducted to verify the reliability of the proposed method. The results show that the proposed algorithm outperforms the available attitude estimation model in motion tracking and is feasible for 3-D human motion capture.

Motion capture technology has been widely used in the sport analysis to improve their performance and reduce the injury risk. Kayak, a popular outdoor sport, employs the coordination of multiple muscles and skeletons, especially those of upper limbs that must be investigated carefully. The fine-time phase segmentation of rowing cycle plays an important role in analyzing kayaker’s technique. Aiming at the problem of laborious manual phase labeling in the traditional video analysis method, an automatic phase segmentation method for kayak rowing is proposed combined with a machine learning algorithm. In this article, inertial sensors and a data fusion algorithm are used to calculate the joint angles between arm and trunk, left elbow and right elbow when the athlete is rowing. According to the permutation and combination principle, the angle sequence is combined in nine different ways, and four machine learning algorithms (decision tree, support vector machine,$k$-nearest neighbor, bagging ensemble learning) are used to study the effects of different combinations on rowing phase division. Among them, the precision of phase segmentation becomes higher with the increase of motion information. The combination of arm to trunk joint angle only needs three data collection nodes; thus, the computational cost is smaller; moreover, all the four algorithms show good classification accuracy (up to 98.1%). The results indicating that the combination of arm to trunk joint angle and support vector machine algorithm could better complete the task of the phase segmentation for kayak rowing.

Horse-rider interaction analysis by wearable sensors is a promising tool for monitoring equestrian training. In this paper, a body sensor network (BSN) based equestrian motion analysis system is developed, which combines bespoke inertial measurement units (IMU) and MindWave electroencephalography (EEG) acquisition equipment. To fuse the mechanical and EEG signals collected from the system, emotional and attitude information can be obtained to analyze the interaction between the rider and horse in equestrian training. For motion data fusion, a novel method, exercise intensity extend kalman filter (EID-EKF), is proposed, which can also reconstruct the riders’ posture in different gaits by establishing a biomechanical model. The accuracy of our method is verified with the optical system Vicon to support the motion capture for four riding styles (walking, sitting troth, rising trot, canter). Finally, the emotion changes of the riders with different levels are quantified, and kinematic analysis is carried out by combining with inertial and emotional information. It is concluded from the experiment results that the estimation errors are well controlled, and motion patterns acquired according to the kinematic analysis are consistent with the actual situation.

New technological innovations are changing the future of healthcare system. Identification of factors that are responsible for causing depression may lead to new experiments and treatments. Because depression as a disease is becoming a leading community health concern worldwide. Using machine learning techniques this article presents a complete methodological framework to process and explore the heterogenous data and to better understand the association between factors related to quality of life and depression. Subsequently, the experimental study is mainly divided into two parts. In the first part, a data consolidation process is presented. The relationship of data is formed and to uniquely identify each relation in data the concept of the Secure Hash Algorithm is adopted. Hashing is used to locate and index the actual items in the data. The second part proposed a model using both unsupervised and supervised machine learning techniques. The consolidation approach helped in providing a base for formulation and validation of the research hypothesis. The Self organizing map provided 08 cluster solution and the classification problems were taken from the clustered data to further validate the performance of the posterior probability multi-class Support Vector Machine. The expectations of the importance sampling resulted in factors responsible for causing depression. The proposed model was adopted to improve the classification performance, and the result showed classification accuracy of 91.16%.

The development of activity recognition based on multi-modal data makes it possible to reduce human intervention in the process of monitoring. This paper proposes an efficient and cost-effective multi-modal sensing framework for activity monitoring, it can automatically identify human activities based on multi-modal data, and provide help to patients with moderate disabilities. The multi-modal sensing framework for activity monitoring relies on parallel processing of videos and inertial data. A new supervised adaptive multi-modal fusion method (AMFM) is used to process multi-modal human activity data. Spatio-temporal graph convolution network with adaptive loss function (ALSTGCN) is proposed to extract skeleton sequence features, and long short-term memory fully convolutional network (LSTM-FCN) module with adaptive loss function is adapted to extract inertial data features. An adaptive learning method is proposed at the decision level to learn the contribution of the two modalities to the classification results. The effectiveness of the algorithm is demonstrated on two public multi-modal datasets (UTD-MHAD and C-MHAD) and a new multi-modal dataset H-MHAD collected from our laboratory. The results show that the performance of the AMFM approach on three datasets is better than the performance of the video or the inertial-based single-modality model. The class-balanced cross-entropy loss function further improves the model performance based on the H-MHAD dataset. The accuracy of action recognition is 91.18%, and the recall rate of falling activity is 100%. The results illustrate that using multiple heterogeneous sensors to realize automatic process monitoring is a feasible alternative to the manual response.

For the purpose of quantitative analysis, this paper proposes a wearable gait analysis method for Parkinson's disease (PD) to evaluates the motor ability. The error state Kalman filter (ESKF) is used for attitude estimation, and the gait parameters are modified by phase segmentation and zero velocity update (ZUPT) algorithm. In addition, this study uses gait parameters as classifier features to recognize abnormal gait, and compares the recognition effect with statistical features. The effect of our gait system is verified by comparison with the OptiTrack system, and the mean absolute error (MAE) of step length and foot clearance are 2.52 ±3.61 cm and 0.96 ±1.24 cm respectively. Forty Parkinson's patients and forty age-matched healthy people are recruited for gait comparison, the analysis results showed significant differences between the two groups. The abnormal gait recognition results show that gait features have stronger generalization ability than statistical features in leave-one-subject-out (LOSO) validation. The method proposed in this study can be applied to the gait analysis and objective evaluation of PD.

Social interactions are frequent and important in daily life. In this paper, we present a multi-featured analysis for recognizing affective actions collected based on body sensor networks. After nonverbal cues that related to emotions are collected from dyadic interactions, we investigate the recognition performance considering different feature dimensions. Besides, recognition accuracy rate of combining different sensor nodes are also evaluated for investigating optimal placements of inertial sensors.

Swimming is a worldwide popular sports whose performance is highly correlated to the posture.To analyze and recognize the posture in swimming, a monitoring system (SwimSense) for human swimming training based on wearable inertial sensors is established. In this paper, one inertial sensor node is arranged on the surface of lumbar, and the raw sensor data concerning four swimming styles was collected. Through data fusion method and statistical analysis, the features of posture and statistical information were extracted. Subsequently, we proposed an action recognition method based on HMM. According to the classification results of different swimming strokes, it can be concluded that our method has high recognition accuracy and certain reference values, which can be used in swimming training in the future.

Gait analysis based on wearable inertial sensors is conceptually well known, which has potential use in various applications related to gait patterns. However, as such systems are inherently nonlinear, their performance characteristics with respective to system settings are loosely researched and poorly documented. Literature in this area mainly focuses on improving system accuracy by fusing complex algorithms and additional sensors, against their gold standard counterparts, such as optical motion capture systems and force plates. This paper addresses the issue of how the system accuracy changes as the parameter settings change, and what system accuracy can be obtained by parameter tuning. A conventional feedforward neural network (FNN) is adopted to detect the temporal gait features that are prerequisite for gait analysis, and a six-phase gait model is adopted to give a close examination of human gait. In general, three main parameters are related to the FNN-based detection algorithm, i.e., the number of network layers, the number of neurons in each hidden layer, and the size of sliding window. The roles of these parameters are analyzed, and their effects on system accuracy is evaluated with multi-subject data, to offer some suggestions for parameter tuning and facilitate the system implementation.

Motion capture based on wearable inertial sensors is a promising technique for swimmers' training. To apply motion capture techniques properly, a swimming motion evaluation method based on inertial motion capture technology is proposed. Our proposed method uses a multisensor data fusion algorithm for swimmers’ attitude estimation, and the swimming posture is reconstructed in combination with a human biomechanical model. Furthermore, a comparative experiment between our proposed motion capture system and the NDI motion tracking system shows that our system performs reliably and accurately, and the estimation errors are well controlled. In addition, the accuracy of the orientation estimation algorithm ranges from$\text{1.65}^{\circ }$to$\text{3.66}^{\circ }$. The system can capture swimmers’ lumbar spine in four competitive swimming styles. A kinematic analysis of lumbar spine movement indicates that the patterns of swimmers’ lumbar spine movement can be used to evaluate training performance and provide quantitative data for swimmers.

Quantitative gait research is an effective auxiliary means to provide effective clinic diagnosis and surgery guidance for patients mobility. The traditional gait analysis carried out in the laboratory cannot be easily applied into the clinical medicine due to its complexity and high operating costs. To overcome this limitation, inertial sensors fixed on the ankle could perform out-of-lab measurements on subjects walking patterns. In this paper, a low-cost and intelligent wearable gait analysis system based on the inertial sensors is proposed, which can measure spatiotemporal gait parameters and help clinicians with diagnosis. The paper solves sensor drift problem by gradient descent method and zero velocity update algorithm. Using the gait analysis system, we conducted the contrast test of the gait parameters between the patients with abnormal gait and normal person. By contrasting two group of volunteers, it could be found that there is a significant difference about most spatiotemporal gait parameters. These results also indicate that our gait analysis system has potential to be used for out-of-lab gait analysis and can effectively detect the gait disorders and provide the reference for clinicians.

Gesture as a natural and efficient interactive mode, which has been widely used in the field of human-computer collaboration, as the present existing gesture acquisition method is difficult to meet the users' immersion experience and ensure the real-time requirements, in this paper, we design a wearable interactive system which can meet the need of real-time hand gesture acquisition and 3D display. From the perspective of human ergonomics, we analysis the relationship between the movements of bones and joints during hand movement and establish a dynamic model about the skeletal structure of hand. On the basis of this theory, combining with the spatial navigation theory and data fusion method of heterogeneous sensors, a hand tree sensor network based on MEMS inertial sensor is established to realize the real-time tracking of gesture. At the same time, we make a comparison and verification of the gesture data by combining with the image processing method through extracting the key frame information in the gesture video. Finally, we can find the system established in this paper can realize the real-time tracking of gestures through analysis and comparison of real gesture, which provides certain reference value.

In this paper, a method of analyzing the equestrian sport based on body sensor network is introduced. This method uses inertial sensors to analysis the motion characteristics of the key parts in the process of equestrian sport, so as to provide references for the correction of equestrian posture. This paper first arranges wireless sensor nodes on the surface of the key parts by analysing the motion of the equestrian sport, then the RAW sensor data can be accepted through 2.4G wireless channel. Next, the three dimensional attitudes of each node are calculated by the gradient descent method, and we compared the estimated attitude with the actual equestrian process. Finally, the experimental results show that using inertial sensors to achieve the motion analysis of equestrian sports has a certain application value. It means that our method can provide some reference for equestrian training.

In this paper, a new method of human motion segmentation is proposed, which the inertial data of human movement was acquired through wearable Inertial measurement unit (IMU), and the feature of raw time series data was directly extracted, which was segmented by sliding window, and then by combining Support Vector Machines (SVM) classifier as the algorithm of motion phase detection. The experimental result shows that the potential pattern of human movement by segmenting the motion phase can be found through pattern recognition technique. The method can be applied into different human movements, such as walking and swimming. The feasibility and effectiveness has been verified.