Multimodal peripheral physiological signal fusion for emotion recognition seeks to perceive or recognize human emotions using peripheral modalities such as electromyography, electrodermal activity, and respiratory wave. Previous approaches to multimodal fusion primarily focus on emotion-sensitive signals such as electroencephalogram (EEG), often overlooking the potential value of peripheral physiological signals in emotion recognition. Moreover, the inherent heterogeneity among different modalities continues to pose challenges to fusion quality. In this article, we propose a multimodal decoupled multimodal fusion (DMF) to address these issues, enabling flexible feature decoupling, cross-modal feature interaction, and relational knowledge learning. Specifically, each modality’s representation is first decoupled into two components: common features and modality-specific features; second, the DMF employs progressive cross attention to facilitate the exchange of modality-specific features across different modalities; and finally, it uses relational knowledge to learn multimodal spliced features, embedding both inter-modal and intra-modal feature relationships. DMF offers a dynamic multimodal emotion recognition framework that leverages the emotional information contained in diverse modalities. Experimental results demonstrate that the DMF method consistently outperforms previous approaches and provides a viable solution for multimodal peripheral physiological signal fusion.

Bradykinesia is a hallmark symptom of Parkinson’s disease (PD) that significantly affects patients’ functional abilities and quality of life. This study proposed a fine-grained classification method for evaluating the level of bradykinesia in PD patients. Based on inertial signals, surface electromyographic (sEMG) signals, and videos obtained from 40 PD patients and 13 healthy subjects, the proposed data preprocessing method extracts 69-D features from inertial and sEMG (IE) signals, and 7-D skeleton features from videos. A two-stream network, including IE stream, skeleton stream, and decision fusion module, was developed using long short-term memory, full convolutional neural networks, and fully connected neural networks. In addition, the IE stream incorporated a feature shrinking module to process high-dimensional features to reduce redundant features. Furthermore, an LSTM-variational autoencoders method was proposed for data augmentation of categories with fewer samples. The proposed method achieved higher recognition rates (pro/supination movements of hands: 85.51%, finger tapping: 88.06%, hand movements: 90.00%) compared to other methods. With low-cost, compact and lightweight methods, bradykinesia in PD patients can be intelligently assessed, which will enhance patient management and treatment efficiency.

The classification analysis of incomplete and imbalanced data is still a challenging task since these issues could negatively impact the training of classifiers, which were also found in our study on the physical fitness assessments of patients. And in fields such as healthcare, there are higher requirements for the accuracy of the generated imputation values. To train a high-performance classifier and pursue high accuracy, we attempted to resolve any potential negative impact by using a novel algorithmic approach based on the combination of multivariate imputation by chained equations and the ensemble learning method (MICEEN), which can solve the two problems simultaneously. We used multivariate imputation by chained equations to generate more accurate imputation values for the training set passed to ensemble learning to build a predictor. On the other hand, missing values were introduced into minority classes and used them to generate new samples belonging to the minority classes in order to balance the distribution of classes. On real-world datasets, we perform extensive experiments to assess our method and compare it to other state-of-the-art approaches. The advantages of the proposed method are demonstrated by experimental results for the benchmark datasets and self-collected datasets of physical fitness assessment of tumor patients with varying missing rates.

Tremors are typically experienced by patients at the beginning of Parkinson's disease (PD). Clinicians evaluate clinical symptoms based on scale and experience, but mild tremors do not have significant characteristics and are difficult to observe with the naked eye. Implementing an intelligent and objective method to identify PD patients with early tremor symptoms and healthy controls (HC) is necessary. This study used wearable sensors to collect 9-axis inertial signals and 2-channel sEMG signals at the wrists of 13 HC and 24 PD patients from Dalian Municipal Central Hospital. Based on a Long Short-Term Memory Network (LSTM), an attention mechanism, and a Fully Convolutional Network (FCN), we develop a model to classify and recognize data from PD patients. Compared the proposed method with several classification methods, the results showed that the proposed method achieved higher classification accuracy (91.78%), precision (100%), recall (87.50%), and F1-score (93.33%) of PD class than Support Vector Machine, FCN, and LSTM. The computing time of the proposed method is approximately 1 second. The proposed method identifies early PD patients by pre-clinical assessment of mild upper limb tremors, which is valuable for early treatment and rehabilitation.

Gait analysis is a prosperous tool for the clinical evaluation and diagnosis. In this article, a portable gait analysis system based on foot-mounted inertial sensors is established. A threshold-free method using a long short-term memory recurrent neural network is constructed to segment four typical gait phases in a gait sequence for the temporal parameters analysis. Segmentation accuracy reaches over 95% across recruited subjects with distinct gait patterns, which is significantly superior when compared with traditional machine learning methods. The zero-velocity indicator is generated successively according to the segmented sequence to accomplish zero velocity update for the spatial parameter calculation. The accuracy of the proposed system is also validated through the OptiTrack in the lab. The comparison result of the stride length shows that the error between the two systems is less than 2%, which demonstrates that our system can satisfy the demand in the clinical.

Using inertial measurement units mounted on foot is a feasible approach to improve the positioning accuracy for the human motion capture system. This paper presents a lightweight and low cost wireless inertial motion capture system for the simultaneous reconstruction of human body attitude and displacement. First of all, the device is based on human sensor networks and distributes 15 sensor nodes on the key human limbs. Then, after an initial sensor alignment with the reduced error, a zero-speed update algorithm is used to calculate foot displacement. In addition, to constantly update the human posture information, a kind of motion reconstruction method based on the gradient descent method was used to fuse the sensor data. Finally, a new method of three-dimensional human body reconstruction is proposed, which is different from the traditional motion capture system. Through unconstrained traversal of the root, the human posture and foot trajectory are combined to realize the synchronous reconstruction of posture and displacement. It is concluded from the experiment results that the estimation errors are well controlled, and motion patterns are consistent with the actual situation.

Wearable inertial motion capture, a new type of motion capture technology, mainly estimates the human posture in 3-D space through multisensor data fusion. The available method for sensor fusion is usually aided by magnetometers to remove the drift error in yaw angle estimation, which in turn limits their application in the presence of a complex magnetic field environment. In this article, an extended Kalman filter (EKF) data fusion method is proposed to fuse the 9-axis sensor data. Meanwhile, the heuristic drift reduction (HDR) method is used to calibrate the accumulated error of a heading angle. In addition, the position in 3-D space is estimated by the foot-mounted zero-velocity-update (ZUPT) technique. Combining 3-D attitude and position, a biomechanical model of the human body is established to track the motion of a real human body. The EKF algorithm and position estimation methods are benchmarked against the golden standard, optical motion capture system, for various indoor experiments. In addition, various outdoor experiments are also conducted to verify the reliability of the proposed method. The results show that the proposed algorithm outperforms the available attitude estimation model in motion tracking and is feasible for 3-D human motion capture.

Motion capture technology has been widely used in the sport analysis to improve their performance and reduce the injury risk. Kayak, a popular outdoor sport, employs the coordination of multiple muscles and skeletons, especially those of upper limbs that must be investigated carefully. The fine-time phase segmentation of rowing cycle plays an important role in analyzing kayaker’s technique. Aiming at the problem of laborious manual phase labeling in the traditional video analysis method, an automatic phase segmentation method for kayak rowing is proposed combined with a machine learning algorithm. In this article, inertial sensors and a data fusion algorithm are used to calculate the joint angles between arm and trunk, left elbow and right elbow when the athlete is rowing. According to the permutation and combination principle, the angle sequence is combined in nine different ways, and four machine learning algorithms (decision tree, support vector machine,$k$-nearest neighbor, bagging ensemble learning) are used to study the effects of different combinations on rowing phase division. Among them, the precision of phase segmentation becomes higher with the increase of motion information. The combination of arm to trunk joint angle only needs three data collection nodes; thus, the computational cost is smaller; moreover, all the four algorithms show good classification accuracy (up to 98.1%). The results indicating that the combination of arm to trunk joint angle and support vector machine algorithm could better complete the task of the phase segmentation for kayak rowing.

Horse-rider interaction analysis by wearable sensors is a promising tool for monitoring equestrian training. In this paper, a body sensor network (BSN) based equestrian motion analysis system is developed, which combines bespoke inertial measurement units (IMU) and MindWave electroencephalography (EEG) acquisition equipment. To fuse the mechanical and EEG signals collected from the system, emotional and attitude information can be obtained to analyze the interaction between the rider and horse in equestrian training. For motion data fusion, a novel method, exercise intensity extend kalman filter (EID-EKF), is proposed, which can also reconstruct the riders’ posture in different gaits by establishing a biomechanical model. The accuracy of our method is verified with the optical system Vicon to support the motion capture for four riding styles (walking, sitting troth, rising trot, canter). Finally, the emotion changes of the riders with different levels are quantified, and kinematic analysis is carried out by combining with inertial and emotional information. It is concluded from the experiment results that the estimation errors are well controlled, and motion patterns acquired according to the kinematic analysis are consistent with the actual situation.

New technological innovations are changing the future of healthcare system. Identification of factors that are responsible for causing depression may lead to new experiments and treatments. Because depression as a disease is becoming a leading community health concern worldwide. Using machine learning techniques this article presents a complete methodological framework to process and explore the heterogenous data and to better understand the association between factors related to quality of life and depression. Subsequently, the experimental study is mainly divided into two parts. In the first part, a data consolidation process is presented. The relationship of data is formed and to uniquely identify each relation in data the concept of the Secure Hash Algorithm is adopted. Hashing is used to locate and index the actual items in the data. The second part proposed a model using both unsupervised and supervised machine learning techniques. The consolidation approach helped in providing a base for formulation and validation of the research hypothesis. The Self organizing map provided 08 cluster solution and the classification problems were taken from the clustered data to further validate the performance of the posterior probability multi-class Support Vector Machine. The expectations of the importance sampling resulted in factors responsible for causing depression. The proposed model was adopted to improve the classification performance, and the result showed classification accuracy of 91.16%.

The development of activity recognition based on multi-modal data makes it possible to reduce human intervention in the process of monitoring. This paper proposes an efficient and cost-effective multi-modal sensing framework for activity monitoring, it can automatically identify human activities based on multi-modal data, and provide help to patients with moderate disabilities. The multi-modal sensing framework for activity monitoring relies on parallel processing of videos and inertial data. A new supervised adaptive multi-modal fusion method (AMFM) is used to process multi-modal human activity data. Spatio-temporal graph convolution network with adaptive loss function (ALSTGCN) is proposed to extract skeleton sequence features, and long short-term memory fully convolutional network (LSTM-FCN) module with adaptive loss function is adapted to extract inertial data features. An adaptive learning method is proposed at the decision level to learn the contribution of the two modalities to the classification results. The effectiveness of the algorithm is demonstrated on two public multi-modal datasets (UTD-MHAD and C-MHAD) and a new multi-modal dataset H-MHAD collected from our laboratory. The results show that the performance of the AMFM approach on three datasets is better than the performance of the video or the inertial-based single-modality model. The class-balanced cross-entropy loss function further improves the model performance based on the H-MHAD dataset. The accuracy of action recognition is 91.18%, and the recall rate of falling activity is 100%. The results illustrate that using multiple heterogeneous sensors to realize automatic process monitoring is a feasible alternative to the manual response.

For the purpose of quantitative analysis, this paper proposes a wearable gait analysis method for Parkinson's disease (PD) to evaluates the motor ability. The error state Kalman filter (ESKF) is used for attitude estimation, and the gait parameters are modified by phase segmentation and zero velocity update (ZUPT) algorithm. In addition, this study uses gait parameters as classifier features to recognize abnormal gait, and compares the recognition effect with statistical features. The effect of our gait system is verified by comparison with the OptiTrack system, and the mean absolute error (MAE) of step length and foot clearance are 2.52 ±3.61 cm and 0.96 ±1.24 cm respectively. Forty Parkinson's patients and forty age-matched healthy people are recruited for gait comparison, the analysis results showed significant differences between the two groups. The abnormal gait recognition results show that gait features have stronger generalization ability than statistical features in leave-one-subject-out (LOSO) validation. The method proposed in this study can be applied to the gait analysis and objective evaluation of PD.

Social interactions are frequent and important in daily life. In this paper, we present a multi-featured analysis for recognizing affective actions collected based on body sensor networks. After nonverbal cues that related to emotions are collected from dyadic interactions, we investigate the recognition performance considering different feature dimensions. Besides, recognition accuracy rate of combining different sensor nodes are also evaluated for investigating optimal placements of inertial sensors.

Swimming is a worldwide popular sports whose performance is highly correlated to the posture.To analyze and recognize the posture in swimming, a monitoring system (SwimSense) for human swimming training based on wearable inertial sensors is established. In this paper, one inertial sensor node is arranged on the surface of lumbar, and the raw sensor data concerning four swimming styles was collected. Through data fusion method and statistical analysis, the features of posture and statistical information were extracted. Subsequently, we proposed an action recognition method based on HMM. According to the classification results of different swimming strokes, it can be concluded that our method has high recognition accuracy and certain reference values, which can be used in swimming training in the future.

Gait analysis based on wearable inertial sensors is conceptually well known, which has potential use in various applications related to gait patterns. However, as such systems are inherently nonlinear, their performance characteristics with respective to system settings are loosely researched and poorly documented. Literature in this area mainly focuses on improving system accuracy by fusing complex algorithms and additional sensors, against their gold standard counterparts, such as optical motion capture systems and force plates. This paper addresses the issue of how the system accuracy changes as the parameter settings change, and what system accuracy can be obtained by parameter tuning. A conventional feedforward neural network (FNN) is adopted to detect the temporal gait features that are prerequisite for gait analysis, and a six-phase gait model is adopted to give a close examination of human gait. In general, three main parameters are related to the FNN-based detection algorithm, i.e., the number of network layers, the number of neurons in each hidden layer, and the size of sliding window. The roles of these parameters are analyzed, and their effects on system accuracy is evaluated with multi-subject data, to offer some suggestions for parameter tuning and facilitate the system implementation.

Human-human interactions are widespread in many fields, but rarely investigated by using body sensor networks (BSNs). Due to the fact that distributed sensors can only provide local information, this paper proposes to analyse the interactional synchrony from both local and overall perspectives. The proposed framework analyses synchrony mainly including windowed cross-correlation, peak picking method and power average operator. Experiments conducted in this paper demonstrate that the proposed framework may not only distinguish between synchrony and asynchrony conditions, but also reveal different situations corresponding to different body parts.

Motion capture based on wearable inertial sensors is a promising technique for swimmers' training. To apply motion capture techniques properly, a swimming motion evaluation method based on inertial motion capture technology is proposed. Our proposed method uses a multisensor data fusion algorithm for swimmers’ attitude estimation, and the swimming posture is reconstructed in combination with a human biomechanical model. Furthermore, a comparative experiment between our proposed motion capture system and the NDI motion tracking system shows that our system performs reliably and accurately, and the estimation errors are well controlled. In addition, the accuracy of the orientation estimation algorithm ranges from$\text{1.65}^{\circ }$to$\text{3.66}^{\circ }$. The system can capture swimmers’ lumbar spine in four competitive swimming styles. A kinematic analysis of lumbar spine movement indicates that the patterns of swimmers’ lumbar spine movement can be used to evaluate training performance and provide quantitative data for swimmers.

Quantitative gait research is an effective auxiliary means to provide effective clinic diagnosis and surgery guidance for patients mobility. The traditional gait analysis carried out in the laboratory cannot be easily applied into the clinical medicine due to its complexity and high operating costs. To overcome this limitation, inertial sensors fixed on the ankle could perform out-of-lab measurements on subjects walking patterns. In this paper, a low-cost and intelligent wearable gait analysis system based on the inertial sensors is proposed, which can measure spatiotemporal gait parameters and help clinicians with diagnosis. The paper solves sensor drift problem by gradient descent method and zero velocity update algorithm. Using the gait analysis system, we conducted the contrast test of the gait parameters between the patients with abnormal gait and normal person. By contrasting two group of volunteers, it could be found that there is a significant difference about most spatiotemporal gait parameters. These results also indicate that our gait analysis system has potential to be used for out-of-lab gait analysis and can effectively detect the gait disorders and provide the reference for clinicians.

In this paper, a method of analyzing the equestrian sport based on body sensor network is introduced. This method uses inertial sensors to analysis the motion characteristics of the key parts in the process of equestrian sport, so as to provide references for the correction of equestrian posture. This paper first arranges wireless sensor nodes on the surface of the key parts by analysing the motion of the equestrian sport, then the RAW sensor data can be accepted through 2.4G wireless channel. Next, the three dimensional attitudes of each node are calculated by the gradient descent method, and we compared the estimated attitude with the actual equestrian process. Finally, the experimental results show that using inertial sensors to achieve the motion analysis of equestrian sports has a certain application value. It means that our method can provide some reference for equestrian training.

In this paper, a new method of human motion segmentation is proposed, which the inertial data of human movement was acquired through wearable Inertial measurement unit (IMU), and the feature of raw time series data was directly extracted, which was segmented by sliding window, and then by combining Support Vector Machines (SVM) classifier as the algorithm of motion phase detection. The experimental result shows that the potential pattern of human movement by segmenting the motion phase can be found through pattern recognition technique. The method can be applied into different human movements, such as walking and swimming. The feasibility and effectiveness has been verified.

This paper presents a monitoring system (Canoe-Sense) for canoe motion based on wearable Body Sensor Networks (BSNs). An effective motion segmentation method was applied to competitive sport, which can segment human motion phases automatically based on raw time series data that was acquired through wearable Inertial Measurement Units (IMUs). Orientation estimation algorithm was adopted to measure the attitude information of athletes' stroke motion of the canoe. By fusing the data of motion phases and attitude changes, the monitoring data may provide coach with a new performance monitoring method for improving coordination motions of two partners or adjusting the training plan in time. The experimental results showed that our system is able to simultaneously monitor motion phases and attitude changes of two athletes during training on the water.

Although abdominal operation is a kind of important clinical surgery, recovery of the post-operative is usually evaluated by the judgment of doctors. This traditional medical care mode may take much time of medical staffs and meanwhile increase the expense of patients. In this paper, a recovery evaluation system is established for the abdominal post-operative by using body sensor networks (BSN). The evaluation system can collect acceleration signals from five parts of human body and surface electro myographic (sEMG)signals from abdomen muscle group (AMG) during a user'sdaily activities. An experiment as a pilot study has been conducted to test the validity and feasibility of the system. The experimental results showed the evaluation system may effectively acquire the relationship between a user's daily activities and intensity of AMG. The study work in this paper may be used as a basis for further study on evaluating recovery level of the actual post-operative.

Contrast media is widely used in hospital for a better imaging of CT angiography. However, emergencies (such as needle eruption and capillary hemorrhage) may exist due to the extreme high speed of injection. A video-object-tracking system is implemented in this paper which can keep track of the patient's progress and sound an alarm in the event of danger. The MeanShift algorithm for real-time object tracking determines the location of the injection site. An area based detection method is presented to check whether the injection site is bleeding. Experiment results show that the video-tracking-and-monitoring system could track the injection site effectively and achieve a high tracking accuracy.