Cross-subject EEG decoding remains a fundamental challenge due to substantial inter-subject variability in brain activity, which hinders the development of subject-independent EEG models. Despite progress in extracting cross-subject invariant features, existing studies neglect the shared neural responses that arise under similar cognitive or emotional states across individuals, limiting their ability to learn generalized and consistent EEG representations. To address the challenges, we propose State Mamba, a novel spatiotemporal EEG state-space model that explicitly models and aligns neural responses and their spatiotemporal state transitions to learn consistent and generalizable representations across subjects. Innovatively, State Mamba theoretically formulates a multi-channel Mamba architecture that jointly models spatial and temporal brain state transitions, supporting principled analysis of neural responses. To enhance spatiotemporal feature coupling, we introduce the LGANN module, which adopts global-local attention to integrate long- and short-term brain activity into a compact EEG representation. Furthermore, we design two self-supervised pretext tasks to extract consistent neural patterns across subjects: (1) representation alignment to align EEG representation, and (2) pattern alignment to align their transition rules under identical conditions, jointly promoting subject-invariant EEG representations. Extensive experiments on three benchmark datasets, FACED, DEAP, and ISRUC, demonstrate the superior performance of State Mamba in cross-subject emotion and sleep recognition tasks, validating its robust generalization capability.

Segmentation of ultra-high resolution (UHR) images is a critical task with numerous applications, yet it poses significant challenges due to high spatial resolution and rich fine details. Recent approaches adopt a dual-branch architecture, where a global branch learns long-range contextual information and a local branch captures fine details. However, they struggle to handle the conflict between global and local information while adding significant extra computational cost. Inspired by the human visual system's ability to rapidly orient attention to important areas with fine details and filter out irrelevant information, we propose a novel UHR segmentation method called Boundary-enhanced Patch-merging Transformer (BPT). BPT consists of two key components: (1) Patch-Merging Transformer (PMT) for dynamically allocating tokens to informative regions to acquire global and local representations, and (2) Boundary-Enhanced Module (BEM) that leverages boundary information to enrich fine details. Extensive experiments on multiple UHR image segmentation benchmarks demonstrate that our BPT outperforms previous state-of-the-art methods without introducing extra computational overhead.

Improving the classification accuracy of multimodal time series data is of great importance in AI-driven healthcare, where temporal physiological signals provide critical insights for disease diagnosis, patient monitoring, and health assessment such as sleep quality evaluation. With the advancement of Large Language Models (LLMs), fine-tuning pre-trained LLMs has become a promising solution to time series classification. However, existing methods often adopt the independent channel processing strategy to avoid negative impact among variables and modalities, neglecting the positive correlations among them. To solve this challenge, we propose a Causality-Enhanced LLM Fine-tuning Method for Multimodal Time Series Classification (CauLLM). CauLLM first calculates the causal weighted matrix of unimodal multivariate data or multimodal data, and then constructs a causal attention mechanism to capture causal correlation features. Ultimately, CauLLM embeds the features into pre-trained LLMs, dynamically adjusting the influence degree of different channels or modalities on the classification results during fine-tuning process. We conduct experiments on both unimodal multivariate datasets and multimodal datasets. Experimental results validate the effectiveness of CauLLM in both unimodal multivariate and multimodal scenarios. The source code of CauLLM is available at https://github.com/ShuangWu111/CauLLM.

In real-world applications, deep neural networks may encounter constantly changing environments, where the test data originates from continually shifting unlabeled target domains. This problem, known as Unsupervised Continual Domain Shift Learning (UCDSL), poses practical difficulties. Existing methods for UCDSL aim to learn domain-invariant representations for all target domains. However, due to the existence of adaptivity gap, the invariant representation may theoretically lead to large joint errors. To overcome the limitation, we propose a novel UCDSL method, called Multi-Prototype Modeling (MPM). Our model comprises two key components: (1) Multi-Prototype Learning (MPL) for acquiring domain-specific representations using multiple domain-specific prototypes. MPL achieves domain-specific error minimization instead of enforcing feature alignment across different domains. (2) Bi-Level Graph Enhancer (BiGE) for enhancing domain-level and category-level representations, resulting in more accurate predictions. We provide theoretical and empirical analysis to demonstrate the effectiveness of our proposed method. We evaluate our approach on multiple benchmark datasets and show that our model surpasses state-of-the-art methods across all datasets, highlighting its effectiveness and robustness in handling unsupervised continual domain shift learning. Codes will be publicly accessible.

Class-incremental learning (CIL) enables models to continuously learn new classes while addressing catastrophic forgetting. With the introduction of pre-trained models, new tuning paradigms have emerged for CIL. This paper revisits parameter-efficient fine-tuning (PEFT) methods in the context of incremental learning. Prior studies reveal that PEFT methods’ extended parameters do not directly contribute to semantic perception, limiting performance with significant category and domain gaps. To address this, we propose semantic-oriented visual prompt learning (SVPL), which enhances semantic perception and improves task-specific knowledge extraction. SVPL assigns learnable prompts to each class, using a contrastive group alignment to align prompts to task-specific semantic spaces, thus preserving relationships between old and new knowledge. Additionally, hierarchical semantic delivery allows the semantic transformation of prompt groups from shallow to deep layers to facilitate efficient knowledge mining and enable effective learning of new knowledge. Extensive experimental results on five benchmarks demonstrate the superior performance of our methods.

Sleep staging is critical for assessing sleep quality and tracking health. Polysomnography (PSG) provides comprehensive multimodal sleep-related information, but its complexity and impracticality limit its practical use in daily and ubiquitous monitoring. Conversely, unimodal devices offer more convenience but less accuracy. Existing multimodal learning paradigms typically assume that the data types remain consistent between the training and testing phases. This makes it challenging to leverage information from other modalities in ubiquitous scenarios (e.g., at home) where only one modality is available. To address this issue, we introduce a novel framework for ubiquitous Sleep staging via Supervised Multimodal Coordination, called SleepSMC. To capture category-related consistency and complementarity across modality-level instances, we propose supervised modality-level instance contrastive coordination. Specifically, modality-level instances within the same category are considered positive pairs, while those from different categories are considered negative pairs. To explore the varying reliability of auxiliary modalities, we calculate uncertainty estimates based on the variance in confidence scores for correct predictions during multiple rounds of random masks. These uncertainty estimates are employed to assign adaptive weights to multiple auxiliary modalities during contrastive learning, ensuring that the primary modality learns from high-quality, category-related features. Experimental results on four public datasets, ISRUC-S3, MASS-SS3, Sleep-EDF-78, and ISRUC-S1, show that SleepSMC achieves state-of-the-art cross-subject performance. SleepSMC significantly improves performance when only one modality is present during testing, making it suitable for ubiquitous sleep monitoring.

Within the domain of smart wearable sensing, facial perception technology can enhance smart glasses by providing more humanized and contextual feedback, thereby playing a crucial role in improving user experience and device interaction. However, the Inertial Measurement Unit (IMU) on glasses faces significant challenges in high-precision facial perception due to unpredictable noise from irrelevant human motion. To address this problem, we propose a Temporal-Aware Domain Adaptation (TADA) approach for noise reduction. Specifically, we introduce Domain Alignment via Class Token (DACT). By leveraging the class token to capture a broader receptive field, DACT enables effective domain alignment through the extraction of global features. Additionally, we design Time Patch Normalization to constrain the model input scale at the feature temporal level. The experimental results demonstrate that our method outperforms multiple methods, highlighting its potential for denoising in high-frame-rate perception tasks. Our code is open-sourced1.

Accurate sleep staging is crucial for wearable sensor-based sleep monitoring and health interventions. Polysomnography (PSG) signals, rich in information from multiple synchronous sensor channels, are frequently utilized in sleep studies due to their high accuracy in sleep staging. However, employing a wearable PSG for sleep monitoring is uncomfortable, complex, expensive, and impractical. The advent of single-channel wearable electroencephalography (EEG) devices enables comfortable sleep monitoring in ubiquitous scenarios (e.g., home). Despite their advantages, such devices lack sufficient information and have severe accuracy limitations. To improve the accuracy of single-channel EEG-based sleep staging, we propose a method called cross-channel association mining (CCAM). Besides the naive feature extraction, CCAM leverages synchronous single-channel EEG and high-density PSG signals to establish pairwise association feature mining models. These association models can transfer information from other auxiliary PSG channels to the target EEG channel. In inference, CCAM exploits association feature mining models to extract effective features from the target EEG channel and improve sleep staging accuracy through directionally multiview information fusion. Experimental results on three public datasets for sleep staging, ISRUC-S1, ISRUC-S3, and sleep heart health study, show that CCAM outperforms state-of-the-art methods, advancing the field of wearable sensor-based sleep monitoring in ubiquitous scenarios.

Deep learning significantly diminishes reliance on human experts for sleep staging, paving the way for fully automated sleep quality assessment in clinical and daily environments. However, existing methods mostly focus on the integration of novel networks through "black boxes" way to boost classification accuracy, often neglecting model interpretability and the identification of critical sleep features. To address these issues, we develop an innovative architecture utilizing Attention-based ensemble Mechanism (namely AeM) for effective sleep staging and further finding out discriminative feature combinations. AeM consists of three main parts, i.e., modality division and combination (MDC) module, individual feature extraction (IFE) module and attention-based weight and ensemble classification (AWEC) module. MDC module divides the raw input data by modality and recombines them along the channel dimension into new inputs. The IFE module employs multiple naive convolutional neural networks to extract features efficiently from each modality combination. Finally, the AWEC module integrates these features, effectively calculates the weight of each modal combination using the attention mechanism, and outputs the classification result. We evaluated AeM on three public-available datasets: ISRUC-S1, ISRUC-S3, and Sleep-EDF78. Experimental results demonstrate that AeM outperforms all comparative methods and can identify effective digital biomarkers in sleep staging tasks.

Assessing motor abilities is crucial for the rehabilitation of post-stroke patients. However, in practical scenarios, medical data often suffer from issues like limited sample size and low data quality. Existing motion recognition methods struggle to learn generalizable features from limited low-quality data, resulting in poor model generalization and low recognition accuracy. Therefore, this paper proposes a spatial-temporal-scale augmentation-based motor ability assessment method (named STSAug). STSAug consists of three parts: a Spatial-Temporal Feature Extractor (STFE), Skeletal Spatial Augmentation (SSA), and Temporal Augmentation based on Contrastive Learning (TA-CL). STFE is designed to extract spatio-temporal features from noisy skeletal point data, mitigating the interference caused by the noise. SSA enhances spatial features of skeletal points that are relevant to classification while minimizing the impact of irrelevant skeletal features on the model. TA-CL applies temporal augmentation and utilizes contrastive learning to enable the model to learn time-independent features, thereby improving the model’s generalization capability. We evaluated our method on the public dataset NTU RGB+D 60 and our own collected dataset of real stroke patient motion data to verify its effectiveness. The experimental results demonstrate that our proposed STSAug possesses a strong ability to extract spatio-temporal generalization features in real scenarios and achieves the best motor ability recognition performance.

Existing non-example class-incremental learning (NECIL) methods usually utilize a combination strategy of replay mechanism and knowledge distillation. However, this combination strategy only focuses on the preservation of old information quantitatively, ignoring the preservation quality. When the old knowledge has wrong redundant information, catastrophic forgetting is more likely to occur. Therefore, obtaining adequate information without impurities as much as possible and removing invalid or even harmful information has become an effective solution to improve the performance of NECIL. This process is consistent with the information bottleneck (IB) theory. Thus, we propose a new NECIL method based on the IB framework. By using the different information obtained from the new and old class samples and the implicit knowledge in the teacher model training process, the error of harmful redundant information learned is eliminated. Specifically, we propose two optimization strategies that align with the two optimization processes of the information bottleneck. Firstly, we employ a pseudo-prototype selection mechanism that selectively incorporates pseudo-samples into the learning process of new and old categories, thus enhancing the distinction between new and old categories and diminishing the mutual information between the input and intermediate features. Secondly, we introduce an attention-based feature distillation method that regulates the distillation strength between feature pairs based on their similarity, thereby augmenting the mutual information between intermediate features and output prediction. Extensive experiments on three benchmarks demonstrate that the proposed method exhibits significant incremental performance improvements over existing methods.

Human activity recognition (HAR) is crucial for health monitoring and disease diagnosis in Internet-of-Things environments. However, existing HAR approaches either suffer from poor accuracy or achieve high accuracy at the expense of costly manual annotations. To overcome the challenge above, we propose a novel method named LLMIE-UHAR that that leverages LLMs and Iterative Evolution to realize Unsupervised HAR. Specifically, with our designed prompt engineering mechanism, we employ large language models to fuse both contextual and semantic information, and annotate key samples selected by a clustering algorithm. Moreover, LLMIE-UHAR enhances the recognition accuracy with iterative evolution of clustering algorithm, large language models and the neural network based recognition model. Experiments conducted on the public ARAS datasets show the efficiency of our method, achieving an accuracy of 96.00%. This highlights the practical value of our approach.

Brain computer interface with time-series physiological signal analysis (e.g., EEG and fNIRS) is commonly-used technology for the auxiliary diagnosis of dementia. However, due to the non-stationary, non-linear and low signal-to-noise ratio of time-series signal, as well as the lack of relevant dementia diagnosis datasets, the discriminative feature learning and model construction with time-series signal face great challenges. Therefore, we proposed an Effective Connectivity-based Multi-view Feature learning (ECMFeat) method to realize the auxiliary diagnosis of dementia with fNIRS signal. ECMFeat uses Dynamic Bayesian Inference and EEGNet to learn the effective connectivity and temporal-spatial features of fNIRS signal, respectively. By the weighted fusion of the features from different views, we construct the final auxiliary diagnosis model. Experiments are conducted on the real clinical environment–the First People’s Hospital of Foshan, with the participation of 25 subjects covering three different cognition groups. Experimental results verify the inter-group differences in effective connectivity and the effectiveness of ECMFeat in diagnosing dementia.

To tackle the high-dimensional data with multiple representations, multi-view unsupervised feature selection has emerged as a significant learning paradigm. However, previous methods suffer from the following dilemmas: (i) They focus on selecting the features that preserve the similarity structure of data, whereas neglecting the discriminative information in the cluster structure; (ii) The orthogonal constraint is often imposed on the pseudo cluster labels, breaking the locality in the cluster label space; (iii) Learning the similarity or cluster structure from all samples is time-consuming. To this end, a Scalable Multi-view Unsupervised Feature Selection with structure learning and fusion (SMUFS) is proposed to jointly exploit the cluster structure and the similarity relations of data. Specifically, SMUFS introduces the sample-view weights to adaptively fuse the membership matrices that indicate cluster structures and serve as the pseudo cluster labels, such that a unified membership matrix across views can be effectively obtained to guide feature selection. Meanwhile, SMUFS performs graph learning from the membership matrix, preserving the locality of cluster labels and improving their discriminative capability. Further, an acceleration strategy has been developed to make SMUFS scalable for large-scale data. An iterative optimization is designed to solve the formulated objective function, and extensive experiments demonstrate the superiority of SMUFS.

In wearable sensor-based human activity recognition (HAR) research, some factors limit the development of generalized models, such as the time and resource-consuming to acquire abundant annotated data, and the inter-dataset inconsistency of activity category. In this paper, we take advantage of the complementarity and redundancy between different wearable modalities (e.g., accelerometers, gyroscopes, and magnetometers), and propose a Modality Consistency-Guided Contrastive Learning (ModCL) method, which can construct a generalized model using annotation-free self-supervised learning and realize personalized domain adaptation with small amount annotation data. Specifically, ModCL exploits both intra-modality and inter-modality consistency of the wearable device data to construct contrastive learning tasks, encouraging the recognition model to recognize similar patterns and distinguish dissimilar ones. By leveraging these mixed constraint strategies, ModCL can learn the inherent activity patterns and extract meaningful generalized features across different datasets. To verify the effectiveness of ModCL method, we conduct experiments on five benchmark datasets (i.e., OPPORTUNITY and PAMAP2 as pre-training datasets, while UniMiB-SHAR, UCI-HAR, and WISDM as independent validation datasets). Experimental results show that ModCL achieves significant improvements in recognition accuracy compared with other SOTA methods.

Sleep stage classification is crucial for sleep state monitoring and health interventions. In accordance with the standards prescribed by the American Academy of Sleep Medicine, a sleep episode follows a specific structure comprising five distinctive sleep stages that collectively form a sleep cycle. Typically, this cycle repeats about five times, providing an insightful portrayal of the subject’s physiological attributes. The progress of deep learning and advanced domain generalization methods allows automatic and even adaptive sleep stage classification. However, applying models trained with visible subject data to invisible subject data remains challenging due to significant individual differences among subjects. Motivated by the periodic category-complete structure of sleep stage classification, we propose a Structure Incentive Domain Adversarial learning (SIDA) method that combines the sleep stage classification method with domain generalization to enable cross-subject sleep stage classification. SIDA includes individual domain discriminators for each sleep stage category to decouple subject dependence differences among different categories and fine-grained learning of domain-invariant features. Furthermore, SIDA directly connects the label classifier and domain discriminators to promote the training process. Experiments on three benchmark sleep stage classification datasets demonstrate that the proposed SIDA method outperforms other state-of-the-art sleep stage classification and domain generalization methods and achieves the best cross-subject sleep stage classification results.

Federated learning (FL) has been an effective way to train a machine learning model distributedly, holding local data without exchanging them. However, due to the inaccessibility of local data, FL with label noise would be more challenging. Most existing methods assume only open-set or closed-set noise and correspondingly propose filtering or correction solutions, ignoring that label noise can be mixed in real-world scenarios. In this article, we propose a novel FL method to discriminate the type of noise and make the FL mixed noise-robust, named FedMIN. FedMIN employs a composite framework that captures local-global differences in multiparticipant distributions to model generalized noise patterns. By determining adaptive thresholds for identifying mixed label noise in each client and assigning appropriate weights during model aggregation, FedMIN enhances the performance of the global model. Furthermore, FedMIN incorporates a loss alignment mechanism using local and global Gaussian mixture models (GMMs) to mitigate the risk of revealing samplewise loss. Extensive experiments are conducted on several public datasets, which include the simulated FL testbeds, i.e., CIFAR-10, CIFAR-100, and SVHN, and the real-world ones, i.e., Camelyon17 and multiorgan nuclei challenge (MoNuSAC). Compared to FL benchmarks, FedMIN improves model accuracy by up to 9.9% due to its superior noise estimation capabilities.

Federated learning (FL) enables multiple clients to collaboratively train a globally generalized model while keeping local data decentralized. A key challenge in FL is to handle the heterogeneity of data distributions among clients. The local model will shift the global feature when fitting local data, which results in forgetting the global knowledge. Following the idea of knowledge distillation, the global model's prediction can be utilized to help local models preserve the global knowledge in FL. However, when the global model hasn't converged completely, its predictions tend to be less reliable on certain classes, which may results in distillation's misleading of local models. In this paper, we propose a class-wise adaptive self distillation (FedCAD) mechanism to ameliorate this problem. We design class-wise adaptive terms to soften the influence of distillation loss according to the global model's performance on each class and therefore avoid the misleading. Experiments show that our method outperforms other state-of-the-art FL algorithms on benchmark datasets.

Surface electromyography (sEMG) is a kind of bioelectric signal of the human body, containing a wealth of action intentions. Among various gesture recognition solutions, sEMG-based solutions show irreplaceable advantages by directly sensing and parsing human muscle activities and converting them into interactive commands. However, sEMG is sensitive to many factors related to users, and there are individual differences among different users. The gesture recognition model trained based on the data of existing users has poor recognition accuracy on the data of new users directly. Excitingly, transfer learning breaks through the independently identical distribution (I.I.D.) assumption of data in different domains, so it shows great potential for cross-user sEMG gesture recognition. Therefore, we propose a Multi-Source Integration based Transfer Learning (MSITL) method to explore cross-user gesture recognition in this paper. MSITL is composed of two main parts, the Source Model Construction Strategy (SMCS) and the Target Model Adaptation Strategy (TMAS). SMCS is a layered integration model. The first layer builds a model for each user. The second layer integrates multiple models through simple majority voting. TMAS is mainly divided into three steps. The first step is to use the target domain data to evaluate the source domain model and obtain the evaluation score of the individual classifier; The second step is to fine-tune the individual classifiers under the guidance of the evaluation scores; The third step is to integrate the adjusted model. Detailed experiments are conducted on benchmark sEMG gesture recognition datasets, including NinaPro (i.e., DB1) and CapgMyo (i.e., DB-a, DB-b, and DB-c). The proposed method achieves significant improvements in performance compared with current state-of-the-art methods.

Federated learning (FL) is a novel distributed learning framework where multiple participants collaboratively train a global model without sharing any raw data to preserve privacy. However, data quality may vary among the participants, the most typical of which is label noise. The incorrect label would significantly damage the performance of the global model. In FL, the inaccessibility of raw data makes this issue more challenging. Previously published studies are limited to using a task-specific benchmark-trained model to evaluate the relevance between the benchmark dataset in the server and the local one on the participants’ side. However, such approaches have failed to exploit the cooperative nature of FL itself and are not practical. This paper proposes a Consensus-based Label Correction approach (CLC) in FL, which tries to correct the noisy labels using the developed consensus method among the FL participants. The consensus-defined class-wise information is used to identify the noisy labels and correct them with pseudo-labels. Extensive experiments are conducted on several public datasets in various settings. The experimental results prove the advantage over the state-of-art methods. The link to the source code is https://github.com/bixiao-zeng/CLC.git .

Human Activity Recognition (HAR) is an important application of smart wearable/mobile systems for many human-centric problems such as healthcare. The multi-sensor synchronous measurement has shown better performance for HAR than a single sensor. However, the multi-sensor setting increases the costs of data transmission, computation and energy. Therefore, the efficient sensor selection to balance recognition accuracy and sensor cost is the critical challenge. In this paper, we propose an Instance-wise Dynamic Sensor Selection (IDSS) method for HAR. Firstly, we formalize this problem as minimizing both activity classification loss and sensor number by dynamically selecting a sparse subset for each instance. Then, IDSS solves the above minimization problem via Markov Decision Process whose policy for sensor selection is learned by exploiting the instance-wise states using Imitation Learning. In order to optimize the parameters of the activity classification model and the sensor selection policy, an algorithm named Mutual DAgger is proposed to alternatively enhance their learning process. To evaluate the performance of IDSS, we conduct experiments on three real-world HAR datasets. The experimental results show that IDSS can effectively reduce the overall sensor number without losing accuracy and outperforms the state-of-the-art methods regarding the combined measurement of accuracy and sensor number.

Discriminating pathologic cognitive decline from the expected decline of normal aging is an important research topic for elderly care and health monitoring. However, most cognitive assessment methods only work when data distributions of the training set and testing set are consistent. Enabling existing cognitive assessment models to adapt to the data in new cognitive assessment tasks is a significant challenge. In this paper, we propose a novel domain adaptation method, namely the Fine-Grained Adaptation Random Forest (FAT), to bridge the cognitive assessment gap when the data distribution is changed. FAT is composed of two essential parts 1) information gain based model evaluation strategy (IGME) and 2) domain adaptation tree growing mechanism (DATG). IGME is used to evaluate every individual tree, and DATG is used to transfer the source model to the target domain. To evaluate the performance of FAT, we conduct experiments in real clinical environments. Experimental results demonstrate that FAT is significantly more accurate and efficient compared with other state-of-the-art methods.

Surface electromyography (sEMG) armband-based gesture recognition is an active research topic that aims to identify hand gestures with a single row of sEMG electrodes. As a typical type of biological signal, sEMG on one channel is nonstationary temporally and related to multiple adjacent muscles spatially, which hinders the effective representation in gesture recognition. To tackle these aspects, we propose a spatial-temporal features-based gesture recognition method (STF-GR) in this article. Specifically, STF-GR first decomposes the nonstationary multichannel sEMG by multivariate empirical mode decomposition, which jointly transforms each channel into a series of stationary subsignals. It can keep the temporal stationarity within-channel as well as the spatial independence across-channel. Then, by the convolutional recurrent neural network, STF-GR extracts and merges spatial-temporal features of decomposed sEMG signal. Finally, a negative log-likelihood-based cost function is used to make the final gesture decision. To evaluate the performance of STF-GR, we conduct experiments on three data sets, noninvasive adaptive hand prosthetic (NinaPro), CapgMyo, and BandMyo. The first two are publicly available, and BandMyo is collected by ourselves. Experimental evaluations with within-subject tests show that STF-GR exceeds the performance of other state-of-the-art methods, including deep learning algorithms that are not focused on spatial-temporal features and traditional machine learning algorithms that use handcrafted features.