This paper considers the challenging problem of 3D Human Pose Estimation (HPE) from a sparse set of Inertial Measurement Units (IMUs). Existing efforts typically reconstruct a pose sequence by either directly tackling whole-body motions or focusing on distinctive spatio-temporal features of local body parts. Unfortunately, these methods ignore existing interdependent motor synergies amongst body parts, which may lead to pose estimation with ambiguous local parts. This observation motivates us to propose a hierarchical learning-based approach, HiPoser, which utilizes a hierarchical shared structure using Mamba blocks as the backbone to focus on the following estimation tasks, involving: 1) torso pose, 2) lower limbs pose, 3) upper limbs pose, and finally 4) global translation. These tasks selectively incorporate body motion states and are to be carried out sequentially in reconstructing part-based poses, which are amalgamated to estimate the final full-body pose with the global translation that satisfies inter-part consistencies. Our hierarchical structure allows HiPoser the flexibility in prioritizing different aspects of pose estimation, to emphasize more on detail or stability. Empirical evaluations over three benchmark datasets demonstrate the superiority of HiPoser over existing state-of-the-art models, suggesting that analyzing the synergistic movement of body parts is indeed important for advancing IMU-based 3D HPE.

Wearable Human Activity Recognition (WHAR) is a prominent research area within ubiquitous computing. Multi-sensor synchronous measurement has proven to be more effective for WHAR than using a single sensor. However, existing WHAR methods use shared convolutional kernels for indiscriminate temporal feature extraction across each sensor variable, which fails to effectively capture spatio-temporal relationships of intra-sensor and inter-sensor variables. We propose the DecomposeWHAR model consisting of a decomposition phase and a fusion phase to better model the relationships between modality variables. The decomposition creates high-dimensional representations of each intra-sensor variable through the improved Depth Separable Convolution to capture local temporal features while preserving their unique characteristics. The fusion phase begins by capturing relationships between intra-sensor variables and fusing their features at both the channel and variable levels. Long-range temporal dependencies are modeled using the State Space Model (SSM), and later cross-sensor interactions are dynamically captured through a self-attention mechanism, highlighting inter-sensor spatial correlations. Our model demonstrates superior performance on three widely used WHAR datasets, significantly outperforming state-of-the-art models while maintaining acceptable computational efficiency.

Imbalanced Time-Series Classification is a critical, yet challenging task across a spectrum of real-world applications. Previous oversampling and generative approaches primarily target the minority class and often rely on static decision boundaries or similarity-based heuristics. However, these methods overlook the underlying causal factors that govern the distinction between majority and minority classes, particularly in scenarios with ambiguous class boundaries. As a result, the generated samples may fail to enhance class separability, thereby limiting improvements in classification performance. To this end, we propose a CounterFactual Augmentation Minority Generation (CFAMG) method based on generative models that aims to discover the causal factors that determine different classes from a causality perspective. Specifically, our method first utilizes a disentangled classifier to distinguish between causal and non-causal factors. Next, we perform counterfactual intervention by replacing the causal factors of majority class samples with those from minority class samples, creating an intervened latent representation that reflects minority characteristics while preserving essential structures. Finally, the trained minority class decoder generates counterfactual minority samples that resemble real minority instances yet remain distinguishable from the original majority class. Extensive experiments demonstrate that our method outperforms state-of-the-art methods in both univariate and multivariate imbalanced time-series classification tasks. The code is published at https://github.com/WangLei-CQU/CFAMG.

A significant challenge in air quality index (AQI) prediction is to accurately evaluate the potential outcomes after conducting interventions in pollutant factors such as industrial emissions for each enterprise. Existed methods often suffer from spurious correlations caused by unmeasured confounders and are lack of interpretability of the model, leading to sub-optimal prediction performance. This motivates us to propose a causal AQI prediction framework (CAP) that employs a structural causal model (SCM) to characterize the causal structural variability of various AQI factors for robust AQI prediction. Specifically, we employ the front-door adjustment to explicitly eliminate unmeasured confounders by intervening in industrial emissions from the target enterprise. Meanwhile, we take industrial emissions of neighboring enterprises into account when intervening in the target enterprise and simulate the dispersion of industrial emissions through a Gaussian plume model based on meteorological factors. Experiments on two real-world datasets validate the superior performance of our model on AQI prediction compared to the state-of-the-art baselines.

A key challenge in sensor-based fall prediction is the fact that a fall event can often occur in various configurations of fall poses together with their own spatio-temporal dependencies. This leads us to define a spatio-temporal model to explicitly characterize these internal configurations of poses. In particular, we introduce a graph neural network with spatio-temporal topological structure to encode such latent relations among poses by capturing representative patterns in fall events. Moreover, a human body orientation estimator is devised to capture human low limbs information, and as a result, separate pose dependencies are globally consistent. Empirical evaluations on two benchmark datasets and one in-house dataset suggest our approach significantly outperforms the state-of-the-art methods.

Cognitive load assessment task faces a significant challenge regarding the neglect of rich spatio-temporal dependencies and causal dependencies in multi-channel physiological signals. To this end, we present a multi-channel spatio-temporal causal representations model that explicitly characterize the inherent causal structural variability and spatio-temporal dependencies within a single channel and interrelationships among multiple channels. Particularly, a causal structure is constructed by optimizing a score-based causal function under the constraint of causal Markov property. It can effectively disentangle the latent spatio-temporal feature variables into two groups: causal representation and task-irrelevant representation. Empirical evaluations on two public datasets and one in-house dataset suggest our model significantly outperforms the state-of-the-art methods.

The primary challenge in cognitive load recognition is the inherent diversity and causality of multivariate physiological changes, as each instance exhibits a distinctive configuration of physiological events and their spatio-temporal causal dependencies. This leads us to define a causal graph designed by prior knowledge about cognitive load to identify the latent factors hidden in the multi-instance bags constructed by the observed instances of multiple physiological channels. In particular, our model introduces the multi-instance causal representation to explicitly disentangle the unique causal configurations of a particular cognitive load state as a variable number of temporal causal variables and spurious causal variables. In addition, GADF maps are constructed to capture the inherent spatio-temporal dependency among multivariate signals in a 2D structural space. A domain adapter is employed to reduce domain bias by effectively transferring the train domain to the test domain in such continuous latent space. Empirical evaluations on two benchmark datasets and two in-house datasets collected by ourselves suggest our model significantly outperforms the state- of-the-art approaches.

Artificial intelligence generated content (AIGC) has emerged as an indispensable tool for producing large-scale content in various forms, such as images, thanks to the significant role that AI plays in imitation and production. However, interpretability and controllability remain challenges. Existing AI methods often face challenges in producing images that are both flexible and controllable while considering causal relationships within the images. To address this issue, we have developed a novel method for causal controllable image generation (CCIG) that combines causal representation learning with bi-directional generative adversarial networks (GANs). This approach enables humans to control image attributes while considering the rationality and interpretability of the generated images and also allows for the generation of counterfactual images. The key of our approach, CCIG, lies in the use of a causal structure learning module to learn the causal relationships between image attributes and joint optimization with the encoder, generator, and joint discriminator in the image generation module. By doing so, we can learn causal representations in image’s latent space and use causal intervention operations to control image generation. We conduct extensive experiments on a real-world dataset, CelebA. The experimental results illustrate the effectiveness of CCIG.

Falls are the leading cause of unintentional human injury , having become a public health event of strong social concern. The fall prediction technology based on wearable inertial sensors is a relatively reliable solution in human activity monitoring, a user scenario with mobility and high information privacy sensitivity, and has the advantages of low cost, small size, and high precision. However, a key challenge in sensor-based fall prediction is the fact that a fall event can often occur in various configurations of fall poses together with their own spatio-temporal dependencies. This leads us to define a spatio-temporal model to explicitly characterize these internal configurations of poses. In particular, we introduce a graph neural network with spatio-temporal topological structure to encode such latent relations among poses by capturing representative patterns in fall events. Moreover, a human body orientation estimator is devised to represent human low limbs information and as a result, separate pose dependencies are globally consistent. Empirical evaluations on two benchmark datasets and one in-house dataset suggest our approach significantly outperforms the state-of-the-art methods.

Deep generative models are effective in style transfer. Previous methods learn one or several specific artist-style from a collection of artworks. These methods not only homogenize the artist-style of different artworks of the same artist but also lack generalization for the unseen artists. To solve these challenges, we propose a double-style transferring module (DSTM). It extracts different artist-style and artwork-style from different artworks (even untrained) and preserves the intrinsic diversity between different artworks of the same artist. DSTM swaps the two styles in the adversarial training and encourages realistic image generation given arbitrary style combinations. However, learning style from single artwork can often cause over-adaption to it, resulting in the introduction of structural features of style image. We further propose an edge enhancing module (EEM) which derives edge information from multi-scale and multi-level features to enhance structural consistency. We broadly evaluate our method across six large-scale benchmark datasets. Empirical results show that our method achieves arbitrary artist-style and artwork-style extraction from a single artwork, and effectively avoids introducing the style image’s structural features. Our method improves the state-of-the-art deception rate from 58.9% to 67.2% and the average FID from 48.74 to 42.83.

A key challenge of oversampling in medical imbalanced data classification is that the generation of new minority samples often neglects rich causal dependencies among features, with each being responsible for disease diagnosis. This leads us to define a constraint-based approach that generates new samples by explicitly discovering and leveraging the inherent local causal variability of features under a global view. Our approach employs causal Markov property to construct a causal network that explicitly characterizes these unique causal configurations of a particular disease as a variable number of nodes and links. By perturbing those learned causal features from majority class, we synthesize new samples in the territory of minority space. An additional sample selection estimator is introduced to choose the most representative samples. Empirical evaluations on four medical datasets suggest our approach significantly outperforms the state-of-the-art methods.

Hand gesture recognition can be exploited to benefit ubiquitous applications using sensors. Currently, the inherent complexity of human physical activities makes it difficult to accurately recognize gestures with wearable sensors, especially in real time. To this end, a real-time hand gesture recognition system is presented in this paper. In particular, sliding window technology and y-axis threshold are used to detect intended gestures from a continuous data stream and then the segmented data are classified by applying a stacked Long Short-Term Memory (LSTM) model. After noise is removed, six-axis sensor data from wrist-worn devices are fed into the model without requiring feature engineering. We use twelve common hand gestures to evaluate the performance of our model. The experimental results demonstrate the feasibility of our proposed system with an accuracy of 99.8% on average. Our approach allows for an accurate and nonindividual hand gesture recognition. It holds potential to be integrated into a smart watch or other wearable devices for intuitive human computer interaction.

The primary goal of skeletal motion prediction is to generate future motion by observing a sequence of 3D skeletons. A key challenge in motion prediction is the fact that a motion can often be performed in several different ways, with each consisting of its own configuration of poses and their spatio-temporal dependencies, and as a result, the predicted poses often converge to the motionless poses or non-human like motions in long-term prediction. This leads us to define a hierarchical recurrent network model that explicitly characterizes these internal configurations of poses and their local and global spatio-temporal dependencies. The model introduces a latent vector variable from the Lie algebra to represent spatial and temporal relations simultaneously. Furthermore, a structured stack LSTM-based decoder is devised to decode the predicted poses with a new loss function defined to estimate the quantized weight of each body part in a pose. Empirical evaluations on benchmark datasets suggest our approach significantly outperforms the state-of-the-art methods on both short-term and long-term motion prediction.

A key challenge in cancer risk prediction is selecting representative features, with each being responsible for cancer diagnosis. This leads us to define a constraint-based approach that employs causal Markov property to discover local causal dependencies between features and cancer risk types. Our approach introduces a causal network generated from an identified network skeleton to explicitly characterize these unique causal configurations of a particular cancer risk as a variable number of nodes and links. It can be analytically shown that the resulting causal network satisfies the causal Markov property, and as a result, all local cause-effect dependencies can be retained and are globally consistent. An additional node selection estimator is introduced to choose the most representative features. Empirical evaluations on four cancer risk datasets suggest our approach significantly outperforms the state-of-the-art methods.

In this paper, we propose RCapsNet, a recurrent capsule network for text classification. Although a variety of neural networks have been proposed recently, existing models are mainly based either on RNN or on CNN, which are rather limited in encoding temporal features in these network structures. In addition, most of these models require to integrate prior linguistic knowledge into them, which is not practical for a non-linguistician to handcraft such knowledge. To address these issues on temporal relational variabilities in text classification, the RCapsNet is presented by employing a hierarchy of recurrent structure-based capsules. It consists of two components: the recurrent module considered as the backbone of the RCapsNet and the reconstruction module designed to enhance the generalization capability of the model. Empirical evaluations on four benchmark datasets demonstrate the competitiveness of the RCapsNet. In particular, it is shown that prior linguistic knowledge is dispensable for the training of our model.

With the rapid development of biomedical technology, discovering causality from genes and human physiological and pathological characteristics has become a hot but challenge spot over the past decades. Due to the increment of the amount of biomedical data, discovering causality from observed data becomes more and more difficult to search this large body of knowledge in a meaningful manner. To address the issues in existing causality discovering models, we introduce a generative Bayesian causal network that combines neural network to explicitly characterize these unique causal-effect relationships as a variable number of nodes and links. Particularly, a basic skeleton is generated for node selection to reduce the network size by minimizing the maximum mean discrepancy among variables. In addition, a causal generative neural network model is presented to construct causal network with cause-effect scores between variables. Empirical evaluations on two publicly available biomedical datasets and four synthetic datasets suggest our approach significantly outperforms the state-of-the-art methods in discovering causal relationships among biomedical variables.

Gesture recognition is ongoing attention in the field of human computer interaction (HCI). With development of deep neural network technology in computer vision, more complex sign languages are possible to recognize but, the research on Chinese language (CSL) recognition remain in discussion. Here we have performed our collected dataset and proposes a new solution to recognize CSL, and further insight on preliminary verification on CSL recognition using 2D image.This paper attempts to reduce the adverse impact of dataset itself on the image recognition network using continuously improved technical method. Present study addresses the following:1) Due to the lack of the CSL image dataset, we made a CSL dataset and used it in the following experiments to verify the usability of the dataset. 2) Using a self-made dataset, we combined the method of hand skeletal gesture recognition to reduce the impact of the gesture overlap and improve recognition accuracy. Finally, a network model was trained and tested on self-made dataset which include some overlapping gestures that are difficult to recognize and achieved the accuracy rate of 0.9324. 3) Put forward the idea of continuing the experiment to improve dataset and using fuzzy semantic recognition for trying to solve the time-domain problem of dynamic sign language recognition which needs linguistic studies.

Hand gesture recognition with wearables typically focuses on the characteristics of a single point on hand, but ignores the diversity of motion information over hand skeleton. As a result, current methods suffer from two key challenges to manage multiple hand joints: displacement detection and motion representation. This leads us to define a spatio-temporal framework, named STGauntlet, that explicitly characterizes the hand motion context of spatio-temporal relations among multiple joints and detects hand gestures in real-time. The framework introduces the Lie algebra to capture the inherent structural varieties of hand motions with spatio-temporal dependencies among multiple joints. In addition, we developed a hand-worn prototype with multiple motion sensors respectively attached to various joints on hand and collected 7000 samples of seven gestures from nine subjects. Our in-lab study shows that STGauntlet is capable of detecting gesture types together with their 3D tracking trajectory with 97.35% and 95.17% accuracies for subject dependent and independent recognition, respectively.

Military sign language is an important form of tactical communication, especially in restrict situations where either distance or a requirement for silence precludes oral means. Unfortunately, when soldiers cannot see each other, the communication mode of tactical gestures is no longer effective, which may hinder military operations. Vision-based approaches have been at the forefront in the field of hand gesture recognition. However, there still lacks of specific datasets and models for the task of military sign language recognition. In this paper, we collected a new first-person dataset named MSL, which contains 16 classes of 3, 840 tactical gesture samples on battle scenario with more than 11, 0000 video frames performed by 10 subjects. Moreover, we present a novel deep network, called ST-Xception architecture, in light of the depthwise separable convolutions to recognize such military sign language. By expanding the convolution filters and pooling kernels into 3D, our network can characterize the inherent spatio-temporal relationship of a certain tactical hand gesture. In particular, we further reduce computational cost and relieve overfitting by replacing the fully connected layers with adaptive average pooling. Experimental results show that our model outperforms existing models both on our in-house MSL dataset and two other benchmark datasets.