Spatial transcriptomics provides unprecedented opportunities to analyze gene patterns while preserving spatial tissue architecture. However, traditional deep learning methods for spatial transcriptomics analysis face significant challenges in multi-modal data integration, spatial dependency modeling, and biological knowledge incorporation, while existing large language models lack explicit spatial modeling capabilities for transcriptomic data. So we first present a Spatial Transcriptomics Embedding with Large Language Models (ST-LLM), a novel simple and effective approach that transforms intricate spatial graph structures into structured textual representations suitable for large language models (LLMs). ST-LLM dynamically constructs graph adjacency construction using reinforcement learning paradigms to adaptively optimize spatial relationships, converts the resulting graphs into hierarchical textual descriptions with spatial context, and leverages pre-trained semantic understanding to generate high-dimensional spatial-aware representations. Comprehensive experiments on 14 datasets demonstrate that ST-LLM achieves comparable or better performance than traditional model. ST-LLM shows that LLMs embeddings provide a new simple and effective path to encoding spatial transcriptomics biological knowledge.

Cytological images originate from exfoliated cells, collected via liquid-based slides and digitized into whole slide images (WSIs). Unlike histological WSIs that exhibit continuous and well-structured tissue, cytological WSIs are sparse in spatial distribution and unstructured in cellular relationships. Typically, the nucleus serves as the primary diagnostic feature, while surrounding cytoplasmic information plays a supportive role. These unique characteristics limit the development of effective foundation models and hinder the transferability of histology-based models for cytopathology. To address this, we propose **Cyto-SSL**, the first self-supervised pretraining framework for cytological images. It introduces **Nuclei-Centered Perturbation**, which highlights individual nuclei by perturbing non-nuclear regions. We also design an SR-Transformer module, which complements this by using sparse attention to concentrate on diagnostically relevant scattered cells, while iRPE helps model to capture local spatial relationships and avoids unnecessary attention to irrelevant global structures. Experimental results show that **Cyto-SSL** enhances performance across diverse cytological datasets and Multiple Instance Learning (MIL) methods. On a WSI-level dataset, it achieved 95.67% accuracy and outperformed ImageNet-pretrained ResNet-50 by 11.33%, demonstrating superior feature representation for cytological analysis. Additionally, **Cyto-SSL** modules are plug-and-play, easily integrated into other pretraining frameworks, yielding a 2.6% accuracy gain across different SSL methods.

Chen Xu, Yu ji, Zhenyu Lv, Yang Yi, Yizhe Yang, Luyao Ji, Chaoyi Chen, Xianyang Wang, Tian Lan, Zhihua Wang, Juan Wang, Xunde Dong, Fuze Tian, Qunxi Dong, Bin Hu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

MOTIVATION: Segmentation of cell bodies and organelles in microscopy images is critical for biological research, particularly in scenarios with multiple regions of interest where spatial continuity is essential. The Mamba architecture, derived from State Space Models (SSMs), has recently gained attention for efficiently modeling long-range dependencies in sequences, achieving excellent results in both natural and medical image segmentation. However, in vision tasks, current Mamba scanning strategies mainly focus on raster-scanning and local-scanning, which introduce spatial discontinuities, severely affecting the effectiveness of segmentation at the pixel level, especially in dense segmentation tasks. RESULTS: In this article, we propose DyMamba, a Mamba-based model featuring a dynamic scanning strategy that adaptively plans scanning paths based on local features and complexity. In addition, to address the challenges of detail prediction and small object detection, we introduce a local aware module that performs pixel-level regional processing on images. DyMamba achieves robust segmentation across diverse microscopy image types, including cell-, organelle- and tissue-scale images. Experiments on six datasets and multiple scanning strategies demonstrate the excellent performance of our method in segmenting microscopy images, achieving an average improvement of 6.9% in mDice and 4.3% in mIoU over state-of-the-art methods across all datasets. AVAILABILITY: The code is released at https://github.com/cbqBit/dymamba.

Critique ability, defined as the capacity to identify and rectify flaws in text generation, is crucial for the applications of Large Language Models (LLMs). As a meta-cognitive capability, enhancing the critique ability of LLMs poses significant challenges. Recent studies have proposed improving this ability through fine-tuning on critique datasets. However, the static data distribution of existing datasets often leads to a mismatch between the training data and the diverse optimization needs of target models, thereby hindering their effectiveness. To address this issue, we introduce a novel Dynamic Iterative Data Distribution Optimization Method (DIDD) that dynamically adjusts training data distributions to align with the specific optimization requirements of target models. Specifically, DIDD detects the vulnerable data distribution of target optimization models by conducting the meta-critique on synthesized test set. The detected vulnerable data distribution are then leveraged to construct the training dataset that aligns with target model more closely, improving the effectiveness of the training dataset. Extensive experimental results across four benchmarks demonstrate that our proposed DIDD effectively alleviates the mismatch between the training dataset and target optimization models.

Electroencephalogram (EEG) individual differences are a critical factor influencing EEG-based emotion recognition, yet they have not been thoroughly investigated, hindering the development of affective Brain-Computer Interfaces (aBCI). Facing the lack of EEG information decoding research, we conducted an interpretable study on EEG individual differences using five datasets (SEED, SEED-IV, SEED-V, RCLS, and MPED). We analyzed the impact of different EEG information (individual, session, emotion, and trial) through sample space visualization, aggregation phenomena quantification, and energy pattern analysis. By examining emotional difference feature distribution patterns, we identified the Cross-Session Consistency of Individual Emotional Patterns (CCIEP) and the Individual Emotional Pattern Difference (IEPD). These characteristics are the main factors impacting emotion recognition stability. To quantify emotional patterns, we proposed the Correction T-test (CT) weight extraction method. Leveraging individual emotional pattern and trial information, we developed the Weight-based Channel-model Matrix Framework (WCMF) to address limitations of traditional modeling approaches caused by IEPD. Finally, WCMF was validated on cross-dataset tasks through two practical scenario experiments. The results demonstrated that WCMF achieves more stable and superior performance compared to traditional methods. This study provides a deeper understanding of EEG individual differences and offers a robust framework to advance aBCI systems.

Multimodal fusion of Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS) has shown great promise in Brain-Computer Interface (BCI) tasks. However, due to differences in physical mechanisms, temporal dynamics, and semantic representations between the two modalities, the fusion process faces significant challenges such as heterogeneity and temporal misalignment. To address this, we propose a cross-modal decoupled deformable distillation (CMD$^{3}$) method, which aims to achieve flexible, efficient, and interpretable EEG-fNIRS fusion learning. CMD$^{3}$first decouples the feature representations of each modality into modality-independent and modality-specific spaces to separately model commonality and complementary information. A deformable feature extraction network is then designed to process shared and specific features individually, enabling cross-modal temporal alignment via predicted dynamic offsets, thereby mitigating response delays between modalities. Furthermore, to facilitate inter-modal knowledge transfer, we construct a dual-space graph distillation module to explicitly migrate semantic information across modalities, with learnable edge weights used to adaptively regulate the distillation strength. CMD$^{3}$is systematically evaluated on public datasets covering emotion recognition and motor imagery tasks. Experimental results demonstrate that CMD$^{3}$consistently outperforms existing fusion approaches in classification performance. Offset visualization further reveals physiologically meaningful temporal attention patterns learned by the model, validating the effectiveness and explainability of the proposed method.

The brain patterns of emotional perception remain a pivotal research domain in affective neuroscience. Modeling the brain as a complex network has become a crucial approach to understanding its functions. However, traditional brain network research based on graph theory primarily focuses on dyadic interactions between brain regions, which cannot effectively characterize the information exchange process among multiple brain regions during emotional cognitive processes. To address these limitations, we shift our perspective from graph theory to the topological data analysis (TDA) of minimal 1-cycles. The 1 cycles or loops within a network represent the fundamental high order interactions in complex networks and serve as essential pathways for information transmission and integration among the distributed networks of brain regions. By focusing on cycle structures in affective brain networks, we propose a novel SMA-EL method based on the collaborative optimization of the Minimal 1-Cycle with Simplicial Maps Annotation (SMA-M1C) method and linear programming, which balances computational efficiency and method performance to reconstruct the optimal cycles in the brain network. This method is applied to the analysis of emotional brain networks in response to positive and negative emotions induced by naturalistic viewing. Comprehensive experiments demonstrate that the 1-cycle structures of the brain's functional patterns exhibit differences at both individual and group levels, aligning with prior research. Furthermore, the 1 cycles we proposed can serve as a biological marker for emotion recognition. These findings may provide new insights into the organization patterns of functional brain networks under diverse emotional states.

Electroencephalogram (EEG) has been widely used for emotion recognition due to its portability and high temporal resolution. It makes success in subject-dependent scenario but faces significant challenges in cross-subject emotion recognition because of non-stationarity of EEG and individual differences. Most previous studies treat all individuals as a single source domain for transferring emotional knowledge, which may introduce irrelevant information and lead to negative transfer. Besides, there is a potential risk that some important information of common emotional features might be ignored. To deal with the issues, we propose a framework called hybrid source selection fusion domain-invariant attention (HSSFDA) for cross-subject emotion recognition. First, source domains are selected by leveraging local and global similarity for knowledge transfer. Then, a specialized attention mechanism is employed to focus on important emotional information extracted from the domain-invariant features. Finally, domain-invariant and domain-specific features are fused to enhance emotion recognition performance. To evaluate the proposed method, experiments are conducted on several public datasets including SEED, SEED_IV, DREAMER and DEAP. The results demonstrate that HSSFDA achieves accuracies of 85.07 %, 72.11 %, 62.36 %, 77.17 %, 58.51 %, and 63.55 % on SEED, SEED_IV, valence and arousal of DREAMER, and valence and arousal of DEAP datasets, respectively, demonstrating competitive performance compared to popular and state-of-the-art methods. Furthermore, we apply the HSSFDA to a self-recorded dataset collected by self-developed three-channel device and validate its effectiveness in practical applications. In conclusion, HSSFDA is a feasible method for cross-subject emotion recognition and has the potential to broaden the application of EEG in the field of affective computing.

Researchers have long sought objective and quantifiable methods for recognizing depression. Electroencephalography (EEG) signals, which reflect brain activities objectively, have emerged as a promising tool for this purpose. However, the practical application of EEG signals faces significant challenges arising from distribution variability across different datasets and subjects. In addition, conventional methods often struggle to effectively capture information related to dynamic transformations in distributions. To address these issues, we propose a Wasserstein distribution-inspired dynamic adversarial network (WDANet) for EEG-based depression recognition. Specifically, WDANet includes a global discriminator that focuses on the marginal distribution of EEG features, a local discriminator that concentrates on the conditional distribution of EEG features, and a Wasserstein distribution discriminator that utilizes Wasserstein distributions derived from various processed EEG features. The experimental results show that WDANet achieved classification accuracies of 83.33%, 75.52%, 73.93%, 76.04%, and 70.94% in cross-subject, cross-dataset experiments conducted on three datasets, demonstrating its effectiveness and superiority compared to state-of-the-art methods. These results support our claim that WDANet enhances the accuracy and interpretability of depression recognition, providing insights and new research directions for the integration of neuroscience and artificial intelligence technologies.

With the rapid advancement of emotion recognition technology, multimodal physiological signals have garnered increasing research attention due to their rich affective representations. However, the substantial heterogeneity across different physiological modalities poses a significant challenge for effective multimodal fusion, limiting the performance of current emotion recognition systems. Moreover, while demographic information inherently encodes valuable emotional cues, its systematic integration into emotion recognition remains underexplored. To address these challenges, we propose R2G$^{3}$Net, a novel hierarchical framework for multimodal emotion recognition. Our model leverages a three-tier architecture: 1) Regional-to-Global Brain Feature Extraction: A BiLSTM-GNN hybrid network hierarchically encodes EEG signals, capturing spatio-temporal patterns from local brain regions to global functional connectivity. 2) Regional-to-Global Cross-Modal Fusion: Peripheral nervous system (PNS) signals are extracted and fused with brain features to enhance physiological representation learning. 3) Regional-to-Global Social Context-Aware Modeling: A hypergraph neural network (HGNN) integrates demographic data to construct dynamic social networks, uncovering higher-order emotional interactions for improved interpretability. Extensive experiments on three benchmark datasets demonstrate R2G$^{3}$Net's superiority in joint spatio-temporal feature learning and social context-aware emotion recognition. Ablation studies and visual analytics further validate that our fused representations outperform state-of-the-art methods in both discriminative capability and model transparency.

Cross-subject electroencephalogram (EEG) emotion recognition is an important task in affective computing, which aims to learn discriminative and generalizable EEG features to reveal emotions across individuals. To achieve robust generalization, existing methods primarily focus on developing domain alignment constraints to learn features that remain consistent across all source domains, while ignoring the potential benefits of domain-specific information in enhancing model discriminability. As a result, these approaches struggle to utilize relevant source domain information to improve predictions for unseen domains. To address this limitation, we propose a novel prompt-guided domain generalization (PGDG) framework that extends the invariance perspective by incorporating domain-specific information through prompt learning. Specifically, the variational autoencoder is first used to extract domain-invariant features, and then the prompt of each source domain is learned by guiding the classifier in the optimal direction. Finally, a multi-head cross-attention mechanism adaptively integrates these domain prompts from multiple source domains to improve the model's generalization ability in target domains. Experimental results on the SEED and SEED-IV datasets show that PGDG outperforms state-of-the-art methods, demonstrating the potential of prompt-guided generalization in improving cross-subject EEG emotion recognition performance.

Cross-subject emotion recognition remains a challenge due to inter-subject variability, which limits the generalized ability of models to unseen subjects. Existing studies commonly rely on tasking-state EEG data from the target subject for adaptation, which requires additional emotion-elicitation experiments and limits practical deployment. Motivated by findings that resting-state EEG can reflect individual-specific neural characteristics, this study proposes a Discriminative Knowledge Fuzzy Transfer Learning Guided by Resting-state EEG (DKFTL-R) for cross-subject emotion recognition without requiring tasking-state EEG data from the target subject. First, resting-state EEG is leveraged to characterize subject-specific neural signatures, by which source-domain selection is informed. Second, an emotion-style projection alignment module is introduced, in which discriminative emotional knowledge and domain-specific style are integrated via adaptive weighting so that a more transferable representation is obtained. Finally, a Takagi-Sugeno-Kang fuzzy classifier is employed to perform fuzzy inference on the transferable representation. Experiments are conducted on DEAP and DENS datasets, where accuracies of 58.79%, 55.89%, 62.91%, and 60.42% are achieved, respectively, demonstrating competitive performance compared with popular and recent baseline methods. To evaluate practical applicability and deployability, the proposed method is conducted on a self-constructed emotion EEG dataset (BHE-EMO), and it achieves 67.00% accuracy for two-class classification and 44.92% for three-class classification tasks, further demonstrating its effectiveness and engineering potential in real-world settings. In conclusion, we propose a new perspective on cross-subject emotion recognition by integrating resting-state EEG information with fuzzy modeling. This study also introduces a new calibration paradigm for affective brain-computer interface systems.

Video object segmentation (VOS) is a fundamental task in video analysis, aiming to accurately recognize and segment objects of interest within video sequences. Conventional methods, relying on memory networks to store single-frame appearance features, face challenges in computational efficiency and capturing dynamic visual information effectively. To address these limitations, we present a Video Decoupling Network (VDN) with a per-clip memory updating mechanism. Our approach is inspired by the dual-stream hypothesis of the human visual cortex and decomposes multiple previous video frames into fundamental elements: scene, motion, and instance. We propose the Unified Prior-based Spatio-temporal Decoupler (UPSD) algorithm, which parses multiple frames into basic elements in a unified manner. UPSD continuously stores elements over time, enabling adaptive integration of different cues based on task requirements. This decomposition mechanism facilitates comprehensive spatial-temporal information capture and rapid updating, leading to notable enhancements in overall VOS performance. Extensive experiments conducted on multiple VOS benchmarks validate the state-of-the-art accuracy, efficiency, generalizability, and robustness of our approach. Remarkably, VDN demonstrates a significant performance improvement and a substantial speed-up compared to previous state-of-the-art methods on multiple VOS benchmarks. It also exhibits excellent generalizability under domain shift and robustness against various noise types.

Fine-grained video captioning aims to generate detailed, temporally coherent descriptions of video content. However, existing methods struggle to capture subtle video dynamics and rich detailed information. In this paper, we leverage preference learning to enhance the performance of vision-language models (VLM) in fine-grained video captioning, while mitigating several limitations inherent to Direct Preference Optimization (DPO). First, we propose a pipeline for constructing preference pairs that leverages the intrinsic properties of VLMs along with partial assistance from large language models, achieving an balance between cost and data quality. Then, we propose Synergistic Preference Optimization (SynPO), a novel optimization method offering significant advantages over DPO and its variants. SynPO prevents negative pReferences from dominating the training, explicitly preserves the model's language capability to avoid deviation of the optimization objective, thus obtains high-quality captions and improves training efficiency by eliminating the need for the reference model. We extensively evaluate our proposed data construction pipeline across three models: AuroraCap, LLaVA1.6-7B-Video and InterVL2-8B. Results demonstrate that our method improve performance in fine-grained video captioning significantly and consistenly. Source code is available at https://github.com/longmalongma/SynPO.

Anxiety disorders (AD) are prevalent psychiatric conditions that profoundly impact adolescent neural development. Abnormal delta-beta cross-frequency coupling (CFC) has been identified as a key electrophysiological marker of altered neural dynamics in individuals with AD. However, most existing studies focus on static analysis within restricted brain regions and predefined frequency bands, which limits the understanding of large-scale dynamic neural communication. Therefore, we propose a novel cross-frequency coupling directed brain network (CFCDBN) framework, which integrates personalized CFC estimation and causal information flow modeling to capture the dynamic interactions of the brain network in AD. Personalized CFC significantly improves the precise representation of AD-related neural dynamics by adaptive frequency band division and individualized oscillation feature extraction, overcoming the limitations of traditional CFC methods. The analysis reveals significant delta-beta coupling abnormalities in the left hemisphere of AD, accompanied by disrupted directional pathways involving the thalamus, precuneus, and insula. These findings suggest impaired emotional and cognitive communication from the subcortical to cortical regions. To validate the efficacy of CFCDBN in distinguishing AD patients from healthy individuals, we developed a direction-aware graph neural network (DA-GNN) model that uses CFCDBN representations as input to capture dynamic neural patterns in causal brain connectivity. Experimental results show that the model consistently outperforms traditional machine learning methods and undirected GNN baselines in automatic AD identification, achieving a classification accuracy of 77.8%, and confirming the value of CFCDBN as a robust biomarker for AD-related network dysfunction. These findings not only deepen our understanding of the neural dynamics underlying AD, but also lay the foundation for personalized and mechanism-driven neuromodulation strategies. The core implementation of the CFCDBN framework is available on GitHub: https://github.com/wdxcjnb6/CFCDBN.

Computer audition-based methods have attracted a great deal of attention in the field of disease detection due to their significant advantages, e.g., non-invasive and convenient operation. Among them, the introduction of information representations inspired by human auditory perception, e.g., Mel-frequency transformation, gives it great potential to approach and even exceed the limits of the human auditory system. However, according to previous research, it remains challenging to fairly assess whether information representations inspired by human auditory perception have a significant positive effect on disease detection. Moreover, performance differences among various information representations and their underlying causes are yet to be thoroughly investigated and analyzed. To this end, we propose an interpretable comparative study on information representations inspired by human auditory perception for disease detection. First, the detection accuracy of different information representations are investigated on two sound datasets (a psychological and a physiological disease) based on the classical model and the proposed Temporal-Spatial Multi-Scale Perception Network. Then, the noise robustness of these information representations are compared by introducing Gaussian noise with varying signal-to-noise ratios (SNRs). Finally, by combining the human auditory perception mechanism and explainable AI techniques, we analyze the reasons for performance differences among various information representations from qualitative and quantitative perspectives. Experimental results demonstrate that information representations inspired by human auditory perception can improve the performance of disease detection with statistical significance. Furthermore, Gammatone Frequency Cepstral Coefficients (GFCCs) outperform other information representations by achieving the highest accuracy, particularly under noisy conditions. The interpretable results further reveal the underlying reasons for GFCC's superior performance, highlighting its ability to capture critical auditory features robustly across varying noise levels.These findings emphasize the potential of auditory perception-inspired representations in advancing computer audition-based disease detection systems and provide a solid foundation for future research in this domain.

Depression is a common and serious mental disorder, characterized by persistent low mood, loss of interest, cognitive dysfunction, and physiological changes. Patients may experience symptoms such as sleep disturbances, changes in appetite, fatigue, and low self-esteem, with severe cases potentially leading to suicidal behavior. There are differences in emotional processing and attention allocation between patients with depression and healthy controls, eye movement characteristics such as fixation patterns, saccade amplitude, and attentional bias have been used as physiological signals for depression detection. Many researchers have developed depression recognition models based on ocular imaging. However, convolutional neural networks, which utilize local receptive fields, can only capture local features in ocular imaging. This paper proposes Multi-Scale Temporal-Frequency Attention Network (MTFNet), which innovatively integrates Multi-Scale time-frequency domain attention into the Video Swin Transformer. Through Multi-Scale Temporal-Frequency Attention Module (MTFAM), MTFNet learns the most important regions in eye movement images, enabling it to capture features more effectively from sequential data and gain a deeper understanding of the structure within eye movement images. Experimental results show that the proposed method achieves a high accuracy of 76.8% on a self-collected eye movement image dataset, outperforming most models. This work provides a novel approach to research on depression recognition based on eye movement images.

Depression remains a leading cause of suicide among college students, highlighting the need for effective and scalable screening methods. Internet usage behavior has shown strong potential for identifying depressive tendencies, but privacy concerns limit its practical use. In this study, we propose a privacy-conscious cross-scale adaptive transformer designed for irregular time series data derived from weakly private online behavior, such as application categories and usage patterns, while excluding content-sensitive or personally identifiable information. Our model incorporates an adaptive sampling strategy to unify temporal resolutions and uses a cross-scale attention mechanism to capture depression-related behavioral patterns. We compared several classic models for irregular time series data, and the proposed method outperformed them, offering a promising, non-intrusive approach for depression detection based on privacy-conscious online activity patterns.

Automatic depression detection technology based on speech and text data is one of the current research hotspots. However, numerous current models may inadequately incorporate the individual characteristics important for improving detection precision. This leads to misjudgment when the model deals with a depressed patient whose behavior significantly differs from the majority of patients. Thus, the goal of this research is to extract individual characteristics from speech and text data among individuals and develop an efficient and stable personalized model for depression detection. We propose a personalized multimodal depression detection model (PMDDM). The proposed model generates personalized embeddings for each individual by utilizing both speech and text features, while learning depressive representations and integrating them with personalized embeddings for joint representations. Lastly, the model receives the joint representations to predict the individual's depression status. We evaluated the proposed model using two datasets related to depression detection, MODMA and MIDD. Compared to the best baseline methods, our method improved the recognition accuracy by 5.88% and 3.84% on the MODMA and MIDD datasets, respectively. The F1 scores increased by 4.55% and 3.45%, respectively. In the cross-dataset generalizability experiments, our method still outperformed the best baseline methods in F1 score and accuracy. Results suggest that the effective integration of personalized information can significantly improve the accuracy and generalizability of depression detection models. Our research provides novel insights into personalized depression detection.

Depression detection using Electroencephalogram (EEG) signals obtained from wearable medical-assisted diagnostic systems has become a well-established approach in the field of affective disorders. However, despite recent advancements, on-board Artificial Intelligence (AI) models still demand substantial computational resources, presenting significant challenges for deployment on resource-constrained wearable medical devices. Embedded Multi-core Processors (MPs) offer a promising solution for accelerating these models. However, the limited computational capabilities of embedded MPs, combined with the structural diversity of AI models, complicate resource allocation and increase associated costs. To address these challenges, we propose a Memory-Aware Multi-Objective Iterative Local Search (MAMILS) algorithm to optimize task scheduling, thereby improving the efficiency of AI model deployment on wearable EEG devices. Experimental results across seven AI models demonstrate that, the MAMILS approach yields substantial improvements in key performance indicators: Total Energy Consumption ($\bm {TEC}$) with an average reduction of 47.57%,$\bm {Makespan}$with an average reduction of 48.75%, and$\bm {Throughput}$with an average increase of 198.37%, all while maintaining satisfactory classification performance for both Machine Learning (ML) and Deep Learning (DL) models. Especially, on-board deployment of EEGNeX achieves an accuracy of 93.4%, sensitivity of 91.6%, and specificity of 95.8%. Further analysis indicates that, when integrated with wearable EEG sensors and executable on-board AI models, the proposed MAMILS optimization strategy shows significant promise in facilitating the widespread adoption of low-power, real-time diagnostic systems for depression detection.

Recent works on remote PhotoPlethysmoGraphy (rPPG) estimation typically use techniques like CNNs and Transformers to encode implicit features from facial videos for prediction. These methods learn to directly map facial videos to the static values of rPPG signals, overlooking the inherent dynamic characteristics of rPPG sequence. Moreover, the rPPG signal is extremely weak and highly susceptible to interference from various sources of noise, including illumination conditions, head movements, and variations in skin tone. To address these limitations, we propose a Physiology-based dynamicity disentangled diffusion (PhysDiff) model particularly designed for robust rPPG estimation. PhysDiff leverages the diffusion model to learn the distribution of quasi-periodic rPPG signal and uses a dynamicity disentanglement strategy to capture two dynamic characteristics in temporal rPPG signal, i.e., trend and amplitude. This disentanglement is motivated by the underlying dynamic physiological processes of vasodilation and vasoconstriction, ensuring a more precise representation of the rPPG signal. The disentangled components are then used as pivotal conditions in the proposed spatial-temporal hybrid denoiser for rPPG reconstruction. Besides, we introduce a periodicity-based multi-hypothesis selection strategy in model inference, which compares the natural periodicity of multiple generated rPPG hypotheses and selects the most favorable one as the final prediction. Extensive experiments on four datasets demonstrate that our PhysDiff significantly outperforms prior methods on both intra-dataset and cross-dataset testing.

Prolonged mental fatigue poses significant risks to both individual health and societal productivity. Traditional methods of detecting mental fatigue often neglect the importance of timely, accessible monitoring. Although blue-enriched light (BEL) is effective at mitigating mental fatigue, prolonged exposure to it can result in visual fatigue and impairment. To address these limitations, we propose a portable fatigue monitoring and BEL feedback system. The system uses three-lead prefrontal Electroencephalogram (EEG) signals to detect mental fatigue in real time and dynamically adjusts BEL exposure based on the user's current fatigue state. This prevents visual issues associated with prolonged BEL exposure. In this work, we develop an onboard executable Convolutional Neural Network (CNN) model using interpretable two-dimensional (2D) convolution implemented with TensorFlow Lite. Our model strikes a favorable balance between classification accuracy (85 %) and computational efficiency, requiring only 50.98 K Floating-Point Operations (FLOPs) and a parameter size of 7.28 KB. When deployed on an EEG sensor, the model operates with just 79.98 KB of RAM and 378.98 KB of ROM. It performs a single inference in 130 ms and consumes 232.31 mW of power per inference. Experimental results demonstrate that the lightweight, onboard executable model integrated with the custom-designed hardware system shows potential for effectively modulating mental fatigue.

Depression has become one of the most prevalent mental health disorders worldwide, highlighting the urgent need for objective and reliable auxiliary diagnostic methods. Electroencephalography (EEG), as a non-invasive technique with high temporal resolution, shows great promise in depression recognition. However, the significant inter-subject variability inherent in EEG signals limits the generalization ability of traditional machine learning and deep learning models in cross-subject scenarios. Although incorporating multi-source data can enhance the representational capacity of transfer learning, it also introduces new challenges, such as distributional conflicts and adaptation strategy inconsistencies between sources, which can lead to negative transfer. To address these issues, we propose a Multi-Source Dynamic Adversarial Adaptation Network (MS-DAAN). This framework constructs independent feature extraction and adversarial adaptation branches for each source domain, incorporates an unsupervised EEG-based source clustering mechanism to form semantically coherent subdomains, and introduces a target-guided source attention module to dynamically weight each source according to its statistical similarity to the target domain. Experimental results demonstrate that MS-DAAN significantly outperforms existing methods across multiple evaluation metrics, validating its effectiveness and robustness in cross-subject EEGbased depression recognition.

Early subthreshold depression identification is critical for intervention but hampered by subjective assessments. Gait analysis offers a promising objective biomarker, but existing computational methods struggle to integrate multi-scale motor patterns and lack the interpretability required for clinical trust. This paper introduces$\mathbf{X}^{\mathbf{2}}$-Gait, a novel deep learning framework designed to address these challenges.$\mathrm{X}^{2}$-Gait pioneers a hierarchical, context-aware mechanism that extracts features at multiple granularities, from fine-grained joint kinematics to holistic body posture. Crucially, it designates macro-level postural features as a “contextual anchor” to dynamically guide the interpretation of micro-level movement features via a gating network. This architecture uniquely enables dual-level interpretability, providing both a high-level postural assessment and specific, corroborating biomechanical evidence for each prediction. Evaluated on a dataset of individuals with subthreshold depression and healthy controls,$\mathbf{X}^{\mathbf{2}}$-Gait not only achieves state-of-theart classification accuracy$(70.26 \%)$but also generates clinically meaningful, hierarchical explanations. This work represents a significant step toward developing transparent, trustworthy, and non-intrusive tools for mental health screening.

Photoplethysmography (PPG) signals have been extensively used for monitoring stress level and improving human mental health. A major obstacle to improving PPG classification is the scarcity of real signals, necessitating the employment of signal synthesis techniques. Incorporating the dynamics of PPG into the generative adversarial network (GAN) helps model the physiological dynamics and improves synthesis quality. However, differential equations that can describe the dynamic characteristics of PPG are unavailable. To address this issue, we propose a novel generative adversarial network for PPG synthesis with data-driven attractor constraints (AC-GAN). Firstly, we design a recurrent neural network (RNN), which can continuously update its hidden states, to extract the chaotic motion characteristics of real PPG signals in a purely data-driven mode. Subsequently, the pre-trained attractor extraction network is used as a prior in the optimization process of a GAN to create PPG signals that conform to the underlying dynamics of physiological systems. Several experiments on three public datasets demonstrate that the signals generated by AC-GAN are optimal in terms of both similarity and usability compared to several state-of-theart methods.

Music therapy has been shown to be effective in treating depression, as supported by numerous clinical studies and randomized controlled trials utilizing subjective reports and psychometric scales. However, a more efficient and objective approach is needed to complement these assessments and to investigate the neural effects of music therapy. This study designed a validity assessment framework based on the dot-probe paradigm, which uses participants' attentional biases toward emotional faces as an objective marker of music therapy effectiveness. To validate this framework, we collected 64 -channel electroencephalogram (EEG) signals from patients undergoing music therapy during a dot-probe task, and explored the dynamic cognitive processes in depressed patients before and after treatment using event-related potentials (ERPs). Our results indicate modulated attentional biases toward negative emotional faces in depressed patients following music therapy. Specifically, we observed shorter response times and higher ERP amplitudes for negative faces compared to positive faces before treatment. These abnormalities were ameliorated after treatment and showed significant correlations with pre- and post-treatment scale measurements. These findings lay a foundation for future artificial intelligence-based systems that could automate the assessment of music therapy effectiveness using neurophysiological markers, potentially enabling personalized treatment approaches and real-time therapeutic adjustments.

Heroin addiction represents a chronic neuropsychiatric disorder characterized by profound alterations in the brain's reward circuitry, particularly within the medial prefrontal cortex (mPFC), ventral tegmental area (VTA), and nucleus accumbens (NAc). Despite mounting evidence that addiction involves dysregulated neural oscillations, frequency domain analysis of reward processing in deeper brain structures remains critically understudied. We collected$\mathbf{6 4}$-channel EEG data from heroin addicts and healthy controls during a monetary incentive delay task with three conditions: positive (potential monetary gain), neutral (no gain or loss), and negative (potential monetary loss). Using advanced source localization techniques with fine realistic head models, we reconstructed brain source signals from key reward circuitry regions (mPFC, VTA, and NAc), analyzing neural responses across three temporal stages: reward anticipation, reward expectation, and reward outcome through event-related desynchronization/synchronization (ERD/ERS) analysis and time-frequency analysis. Heroin addicts exhibited significantly altered neural oscillatory patterns compared to healthy controls across delta, alpha, beta, and gamma frequency bands within the reward circuitry, with frequency-specific abnormalities observed in both cortical and subcortical reward-related regions during different stages of reward processing. This study provides the first comprehensive characterization of frequency-specific neural dysfunction spanning the entire reward circuitry in heroin addiction, offering novel insights into the oscillatory mechanisms underlying reward processing abnormalities and informing the development of frequency-targeted therapeutic interventions.

Recently, the electroencephalogram (EEG) has been widely adopted as a quantitative indicator for monitoring emotional modulation during virtual reality (VR) experiences. Although VR emotion-induction materials are continuously being developed, few methods have been proposed for constructing VR scenarios through psychophysiological calibration. To achieve precise emotional regulation, we propose an integrated psychophysiological oriented EEG-based VR scenario modeling approach for emotion induction. This methodology constructs 3D VR scenarios by extending 2D elements through core processes: (1) deconstruction of emotion-annotated 2D images to extract visual-audio emotional patterns, (2) development of immersive environments with dynamic camera trajectories, and (3) integration of audio stimuli and real-world physical configurations to form multi-sensory emotional stimuli. During emotioninduction experiments, EEG signals reflecting emotional states were captured and analyzed for each scenario. We implemented multidimensional calibration of emotion-induction materials by: (a) calculating emotion indicators from EEG signals, (b) combining these with traditional assessments using the Self-Assessment Manikin (SAM) scale, and (c) calibrating scenarios for distinct emotion categories. This yields psychophysiologically calibrated VR scenarios for emotion induction, delivering standardised affective stimuli with synchronised EEG datasets to advance affective computing research.

The task of knowledge graph entity typing (KGET) aims to infer the missing types for entities in knowledge graphs, which is a significant subtask of knowledge graph completion (KGC). In despite of its progress, we observe that the sparsity of the dataset greatly affects the task itself as well as downstream tasks. In this paper, we propose a framework to alleviate this problem, which consists of data augmentation and type inference. We introduce a Statistics-based Entity Type Data Augmentation (SET-DA) method in data augmentation phase, which calculates a type probability distribution for each entity by statistically determining the global relation-type statistics information, and then employ embedding-based KGET models for type inference. Experimental results on two widely used datasets indicate that the proposed framework can solve the data sparsity problem in the task of KGET and the performance of models trained with data augmentation by SET-DA significantly outperforms previous state-of-the-art methods.

Long video understanding with Large Language Models (LLMs) enables the description of objects that are not explicitly present in the training data. However, continuous changes in known objects and the emergence of new ones require up-to-date knowledge of objects and their dynamics for effective understanding of the open world. To alleviate this, we propose an efficient Retrieval-Enhanced Video Understanding method, dubbed REVU, which leverages external knowledge to enhance the performance of open-world learning. First, REVU introduces an extensible external text-object memory with minimal text-visual mapping, involving static and dynamic multimodal information to help LLMs-based models align text and vision features. Second, REVU retrieves object information from external databases and dynamically integrates frame-specific data from videos, enabling effective knowledge aggregation to comprehend the open world. We conducted experiments on multiple benchmark datasets, and our model demonstrates strong adaptability to out-of-domain data without requiring additional fine-tuning or re-training. Experiments on benchmark video understanding datasets reveal that our model achieves state-of-the-art performance and robust generalization.

Wearable Internet of Things (IoT) devices are widely used in many fields. Such as smart bracelets for health, smart watches for sports, smart safety helmets for industry, and so on. These devices make life easier and more efficient. Multiple-input-multiple-output (MIMO) antenna technology improves data transmission rates and communication reliability by using multiple antennas in a device, so as to meet the demand for real-time data transmission and multidevice connectivity in the IoT. A compact tri-port MIMO antenna with high isolation suitable for wearable applications is proposed. The antenna is designed to work in the frequency band of 2.45 GHz industrial, scientific, and medical (ISM). The antenna has two layers and three ports with low profile suitable for wearable applications. The upper layer is a circular patch with four shorting pins, which excites the${\mathrm { TM}}_{01}$mode with a conical radiation pattern. The lower layer is a circular patch inlaid in an annular patch, which respectively excite the${\mathrm { TM}}_{11}$mode and the${\mathrm { TM}}_{31}$mode with a broadside radiation pattern. To achieve high isolation, characteristic mode analysis (CMA) was conducted on the circular patch, the circular patch with four shorting pins, and the annular patch. The positions of the three ports are reasonably arranged based on mode diversity. The measured bandwidth of the port for conical radiation is 35 MHz (2.43–2.465 GHz), while the measured bandwidths of the two ports for broadside radiation are basically the same which is 50 MHz (2.417–2.467 GHz). The measured overall isolation exceeds 50 dB.

To address the growing challenge of energy efficiency in next-generation coordinated multipoint (CoMP) communication systems, this article develops a green CoMP optimization framework that integrates renewable energy harvesting, smart grid interactions, and real-time power control. We formulate a stochastic long-term weighted sum-rate maximization problem, incorporating transmit covariance variables and joint channel-aware precoding. To enable online implementation, we convert the time-averaged problem into an equivalent per-slot formulation and design an online dynamic beamforming and energy management (ODBEM) algorithm. The proposed ODBEM integrates three synergistic mechanisms: 1) dual-driven energy pricing; 2) Lyapunov drift-plus-penalty scheduling; and 3) momentum-based energy smoothing. We further conduct rigorous convexity and Karush–Kuhn–Tucker optimality analysis to ensure algorithmic correctness and convergence. Simulation results demonstrate that ODBEM outperforms baseline strategies in both throughput and energy cost, confirming its effectiveness for sustainable and adaptive CoMP transmission.

Objective depression assessments based on physiological data offer new opportunities for clinical diagnostic support, but their usage is constrained by the limitations of data collection devices. Ubiquitous digital devices, now deeply integrated into everyday life, have the potential to effectively capture and represent digital phenotypes of depression. However, existing research in this field faces fundamental problems, such as device thresholds and insufficient utilization of distributed computational power, which have hindered significant research and application efforts in this area. In this article, we proposed the Computation-Oriented Hierarchical Depression Detection Internet of Things (IoT) Framework, which allows IoT devices to collaborate in a layered and distributed manner for psychological data collection and depression detection. In addition, we developed a depression detection model, i.e., spike memory transformer (SMT), which significantly reduces inference energy consumption, facilitating the deployment of depression detection capabilities across various IoT terminal devices. Experimental results demonstrate that our model achieved up to 70% accuracy on the D-Vlog dataset while reducing inference power consumption by an average of 38% compared to classical deep learning methods, thus validating the feasibility of proposed method.

Large AI models, connected to terminal devices via high-speed mobile communication networks, enable task collaboration and resource sharing, forming an intelligent framework for the Symbiotic Internet of Things (SIoT) paradigm in industry applications. Despite large language models hold significant potential for psychological intervention, their emotional interaction capabilities remain limited. This paper introduces a SIoT framework for psychological intervention and proposes an emotion-enhanced human-machine interaction architecture incorporating behavioral information captured by IoT devices. The system leverages ubiquitous sensing devices such as cameras and microphones, along with technologies like speech recognition and generation, as well as hyper-realistic digital humans, to create a natural interaction interface. Additionally, we introduce the end-of-utterance detection method and the behavior pattern control algorithm to facilitate smoother and more goal-oriented conversations. The proposed methods and prototype system have been validated through subjective and objective experiments, with results demonstrating their feasibility and suggesting that this approach could become one of the primary forms of humanmachine interaction for psychological intervention in the future.

Colorectal Cancer (CRC) is one of the leading causes of cancer-related deaths worldwide, and early screening plays a crucial role in improving patient outcomes. In this study, we present a novel AI-assisted CRC diagnostic system using Bowel Sound (BS) signals. We first develop two portable BS acquisition devices with distinct form factors for high-fidelity signal capture in both clinical and home-care scenarios. A total of 221 recordings were collected under expert-guided protocol, with 144 CRC recordings and 59 Non-CRC healthy controls using the developed device. To enable low-resource deployment, we design a lightweight deep learning model optimized for real-time, on-board inference. The model incorporates multiple training strategies, including transfer learning on a large-scale public BS dataset, self-supervised temporal feature learning, and a hybrid semi-and weakly-supervised approach that leverages both unlabeled and real-noise data. Furthermore, a Sound Event Detection (SED) attention mechanism and iterative consistency learning are introduced to enhance the model’s sensitivity to BS activity. The proposed model comprises only 264.7 K parameters and 253.2 M Floating-Point Operations (FLOPs), requiring 1.57 MB of RAM and 1.03 MB of FLASH when deployed on microcontroller. It performs inference in approximately 3.4 s with low power consumption, making it well-suited for low-resource environments. Despite its compact design, the model achieves 93.06% classification accuracy, 96.46% sensitivity, and 86.99% specificity for binary-classes in CRC diagnosis. These results demonstrate the system’s potential for accessible and cost-effective CRC screening in community, home, and rural healthcare scenarios.

In noninvasive brain-computer interfaces (BCIs), EEG analysis plays a critical role, with neural networks serving as a cornerstone for signal decoding. Existing neural network approaches for EEG signal recognition require extensive manual design and hyperparameter tuning, leading to inefficiencies and making them impractical for embedded devices due to their large model size. To address these limitations, we propose mixed-level differentiable and hardware-aware neural architecture search (MDH-NAS), a framework that automatically generates lightweight neural networks tailored for EEG classification. Unlike traditional DARTS methods, MDH-NAS employs a hybrid optimization strategy that balances global and local search spaces, thereby accelerating and refining architecture discovery. It introduces explicit size constraints during the search process to ensure deployability on embedded devices. MDH-NAS demonstrates autonomous generation of architectures for tasks such as motor imagery (MI) and depression recognition, achieving 87.80% accuracy on the BCI-IV dataset and 90.09% on the MODMA dataset. When deployed on the EAIDK-610 board across heterogeneous tasks, it attains 85.37% accuracy on the EEG Motor Movement/Imagery dataset. This method reduces architecture discovery time by 89% and enhances prediction accuracy by 8.70% compared to baseline methods, highlighting its potential for scalable EEG analysis and real-world embedded deployment.

Speech contains rich information on the emotions of humans, and Speech Emotion Recognition (SER) has been an important topic in the area of human-computer interaction. The robustness of SER models is crucial, particularly in privacy-sensitive and reliability-demanding domains like private healthcare. Recently, the vulnerability of deep neural networks in the audio domain to adversarial attacks has become a popular area of research. However, prior works on adversarial attacks in the audio domain primarily rely on iterative gradient-based techniques, which are time-consuming and prone to overfitting the specific threat model. Furthermore, the exploration of sparse perturbations, which have the potential for better stealthiness, remains limited in the audio domain. To address these challenges, we propose a generator-based attack method to generate sparse and transferable adversarial examples to deceive SER models in an end-to-end and efficient manner. We evaluate our method on two widely-used SER datasets, Database of Elicited Mood in Speech (DEMoS) and Interactive Emotional dyadic MOtion CAPture (IEMOCAP), and demonstrate its ability to generate successful sparse adversarial examples in an efficient manner. Moreover, our generated adversarial examples exhibit model-agnostic transferability, enabling effective adversarial attacks on advanced victim models.

Facial expressions in videos inherently mirror the dynamic nature of real-world facial events. Consequently, facial expression recognition (FER) should employ a dynamic graph-based representation to effectively capture the relational structure of facial expressions rather than relying on conventional grid or sequence methods. However, existing graph-based approaches have their limitations. Frame-level graph methods provide a coarse representation of the facial graph across time and space, while landmark-based graph methods need to introduce additional facial landmarks, resulting in a static graph structure. To address these challenges, we propose spatial-temporal relation-aware dynamic graph convolutional networks (ST-RDGCN). This fine-grained relation modeling approach enables the dynamic modeling of evolving facial expressions in videos through dynamic space-time graphs, eliminating the need for facial landmarks. ST-RDGCN encompasses three graph construction paradigms: dynamic independent space graph, dynamic joint space-time graph, and dynamic cross space-time graph. Furthermore, we propose a relation-aware space-time graph convolution (RSTG-Conv) operator to learn informative spatiotemporal correlations in dynamic space-time graphs. In extensive experimental evaluations, our ST-RDGCN demonstrates state-of-the-art performance on the five popular video-based FER datasets, achieving overall accuracy scores of 99.69%, 91.67%, 56.51%, 69.37%, and 49.03% on the CK+, Oulu-CASIA, AFEW, DFEW, and FERV39k datasets, respectively. In particular, our ST-RDGCN outperforms the current best method by 3.6% in UAR on the most challenging FERV39k dataset. Furthermore, our analysis reveals that the dynamic cross space-time graph scheme is the most effective among the three dynamic graph construction schemes.

College students face increasing stress from difficulties with studies, employment, and social interactions, which, if left unaddressed, may lead to depression and physical illnesses. Currently, the detection of stress severity relies on self-assessment scales, while machine learning or deep learning-based approaches primarily focus on classification. This study proposes an approach using pulse signals containing emotional cues and deep learning to automatically detect the severity of stress in college students. First, pulse signals of 177 college students were collected using photoplethysmography (PPG) during they watched five virtual reality (VR) emotional scenes, including calm, sadness, happiness, fear, and tension. Pulse rate variability (PRV) and discrete PPG (dPPG) were extracted from these signals as input for detecting stress severity. Then, the proposed stress detection framework, 1DCNN-BiLSTM + Cross-Attention + XGBoost, was employed to detect stress severity, incorporating an emotional Cross-Attention mechanism. The impact of induced emotions on stress severity detection performance was examined. The results indicated that stress severity detection in emotional scenes outperformed in calm. Furthermore, the detection performance that integrates multiple emotions surpassed single emotions. The fusion of PRV and dPPG signals yielded the best detection performance. This study provides an end-to-end automated approach for detecting stress severity in college students.

EEG emotion recognition presents several challenges, including region correlation, local and long-range node connectivity, and multi-channel patterns, necessitating advanced methods capable of effectively capturing and utilising complex EEG signal information. This paper introduces a novel method, the dual-gate Chebyshev Linkage network (DGC-Link), which comprises three main components: the Chebyshev Linkage (CL) module for extracting regional correlation features, the dual-gate module for regulating the flow of different-order information, and the deep network for extracting multi-channel features and enhancing representation capabilities. Validated on three datasets (SEED, DREAMER, and MPED) with ablation experiments demonstrating each component's effectiveness, DGC-Link achieves superior recognition performance compared to state-of-the-art methods. Notably, it achieves 96.43% accuracy on differential entropy on the SEED dataset, and 98.58%, 97.62%, and 98.01% for valence, arousal, and dominance classifications on the DREAMER dataset, along with 78.48% and 44.93% for 3-class and 7-class classifications on the MPED dataset. These results highlight DGC-Link's potential for improved performance in EEG emotion recognition.

Facial depression recognition is one of the current hot topics. Mainstream methods mainly focus on how to design deep models to effectively extract the difference in facial movements between depressed patients and healthy people. However, this difference changes when the stimulus source to which the subjects are exposed changes. This leads to the performance degradation in cross-stimulus situation and limits the practical application of this technology. We hold the opinion that why depressed patients show behavioral characteristics different from healthy people is that they have a specific stable pattern of responding to stimulus. Therefore, we incorporate stimuli into the modeling process for the first time and employ deep networks to learn stable representations between stimulus and response to achieve stable and effective modeling. Specifically, we propose a deep modeling framework to learn the stimulus-response pattern of the subject through the interaction relationship between the stimulus videos and the subject’s facial movements. We constructed a balanced depression dataset of 364 individuals with three different stimulus videos to verify the effectiveness of our method. The results show that our method achieves state-of-the-art and the best generalization performance in depression recognition. This stimulus-response pattern modeling provides a new perspective for recognizing depression.

Feature selection is a crucial step in EEG emotion recognition. However, it was often used as a single objective problem to either reduce the number of features or maximize classification accuracy, while neglecting their balance. To address the issue, we proposed Improved Multi-objective Grey Wolf Optimization Feature Selection (IMGWOFS). First, we designed a population initialization operator via discriminability and independence of features to accelerate search speed. Second, we employed a two-stage update strategy to improve the global search capabilities of the EEG feature subsets. Finally, we incorporated an adaptive mutation operator to escape the local optima. We conducted experiments on SEED and DEAP datasets, and the accuracy were 86.87$\pm$1.62 % and 60.65$\pm$1.51 % in the beta band using a smaller number of EEG features. In addition, the frontal lobe was related to emotion processing. In conclusion, IMGWOFS is an effective and feasible feature selection method for EEG-based emotion recognition.

Mental disorders have increased rapidly and have emerged as a serious social health issue in the recent decade. Undoubtedly, the timely treatment of mental disorders is crucial. Emotion regulation has been proven to be an effective method for treating mental disorders. Music therapy as one of the methods that can achieve emotional regulation has gained increasing attention in the field of mental disorder treatment. However, traditional music therapy methods still face some unresolved issues, such as the lack of real-time capability and the inability to form closed-loop systems. With the advancement of artificial intelligence (AI), especially AI-generated content (AIGC), AI-based music therapy holds promise in addressing these issues. In this paper, an AIGC-based closed-loop music intervention system demonstration is proposed to regulate emotions for mental disorder treatment. This system demonstration consists of an emotion recognition model and a music generation model. The emotion recognition model can assess mental states, while the music generation model generates the corresponding emotional music for regulation. The system continuously performs recognition and regulation, thus forming a closed-loop process. In the experiment, we first conduct experiments on both the emotion recognition model and the music generation model to validate the accuracy of the recognition model and the music quality generated by the music generation models. In conclusion, we conducted comprehensive tests on the entire system to verify its feasibility and effectiveness.

The heterogeneity has been recognized as a large obstacle to the treatment of autism spectrum disorder (ASD). Recent studies have identified several subgroups of ASD that exhibited heterogeneous alterations in brain. However, most of them primarily depicted the pairwise similarity between individuals, relying solely on a single imaging modality. This leads to an underestimation of the complexity in inter-individual relationships and the rich information provided by multimodal images. To capture the high-order relationships among individuals, we utilized multi-task method to construct multilayer hypergraph based on brain structure and function. We then developed a novel co-optimized community detection algorithm, which jointly optimizes the modular structure across hypergraph layer, with the aim of categorizing subtypes of ASD by fusing multimodal information. By applying the proposed method on the Autism Brain Imaging Data Exchange repository data (n = 287/303, ASD/typical development [TD]), we identified two ASD subtypes with distinct alteration patterns in both brain structure and function. Distinct clinical manifestations in social and communication were observed between the two subtypes. Furthermore, subtyping significantly enhanced the diagnostic accuracy of ASD by over 10%. In addition, our method exhibited superior clustering performance that outperformed traditional community detection algorithms on graphs. Taken together, our study demonstrated the effectiveness of subtyping ASD through a multimodal multilayer hypergraph, highlighting its potential in elucidating the heterogeneity of autism and improving clinical diagnosis.

The advancement of wearable electronic technology has facilitated the integration of smart wearable devices into artificial intelligence (AI)-driven medical assisted diagnosis. Embedded multi-core processors (MPs) have gradually emerged as pivotal hardware components for smart wearable medical diagnostic devices due to their high performance and flexibility. However, embedded MPs face the challenge of balancing performance, power consumption, and load-balancing. In response, we introduce a Pareto-based iterated local search (PILS) algorithm for task scheduling, which systematically optimizes multiple objectives, alongside a task list model to reduce the dimension of the decision space and enhance scheduling performance. In addition, we present a two-stage discretization scheme to ensure that the proposed algorithm offers meaningful guidance throughout the scheduling process. Simulation and on-board testing results show that the proposed algorithm effectively optimizes energy consumption, task execution time, and load balancing in embedded MPs task scheduling, indicating the potential of the proposed algorithm in enhancing the performance of smart wearable medical diagnostic devices powered by embedded MPs.

Cryo-electron microscopy single particle analysis (cryo-EM SPA) is the most powerful technique for biomacromolecule structure determination. However, many factors such as complicated noise and radiation damage make the quality of cryo-EM images extremely poor, where high-frequency structure details are submerged, limiting the application of deep learning and suppressing the resolution of reconstruction. Thus, image restoration is of vital importance. Some related works explore micrograph restoration, but the particles in restored micrographs are still of poor quality. Moreover, the training approach of existing methods uses noisy observations or simulated data as supervision, leading to reduced performance on real cryo-EM data. In this paper, we define the task of particle restoration and propose a novel 4-step framework to this end. Labels are created for each particle image and paired data is collected within our framework, compensating for the absence of ground truth. A deep neural network with encoder-decoder architecture is designed to learn the mapping from degraded particles to high-quality ones, while other networks can also be employed as a plug-and-play module. Three datasets are constructed from real cryo-EM data and extensive experiments are carried out. Both quantitative metrics and qualitative visualization indicate that our framework is effective for cryo-EM particle restoration. It becomes easier to extract particle features after restoration, aiding in SPA and the effective application of deep learning on cryo-EM images. The downstream task experiments of cryo-EM SPA are also conducted, showing that the proposed framework has the potential to improve cryo-EM SPA performance.

Vision-based depression estimation is an emerging yet impactful task, whose challenge lies in predicting the severity of depression from facial videos lasting at least several minutes. Existing methods primarily focus on fusing frame-level features to create comprehensive representations. However, they often overlook two crucial aspects: 1) inter- and intra-cue correlations, and 2) variations among samples. Hence, simply characterizing sample embeddings while ignoring to mine the relation among multiple cues leads to limitations. To address this problem, we propose a novel Multi-Cue Contrastive Learning (MCCL) framework to mine the relation among multiple cues for discriminative representation. Specifically, we first introduce a novel cross-characteristic attentive interaction module to model the relationship among multiple cues from four facial features (e.g., 3D landmarks, head poses, gazes, FAUs). Then, we propose a temporal segment attentive interaction module to capture the temporal relationships within each facial feature over time intervals. Moreover, we integrate contrastive learning to leverage the variations among samples by regarding the embeddings of inter-cue and intra-cue as positive pairs while considering embeddings from other samples as negative. In this way, the proposed MCCL framework leverages the relationships among the facial features and the variations among samples to enhance the process of multi-cue mining, thereby achieving more accurate facial depression estimation. Extensive experiments on public datasets, DAIC-WOZ, CMDC, and E-DAIC, demonstrate that our model not only outperforms the advanced depression methods but that the discriminative representations of facial behaviors provide potential insights about depression. Our code is available at:https://github.com/xkwangcn/MCCL.git

Facial movement-based depression recognition has garnered considerable attention in automatic depression detection field in recent years. Current methods typically seek to establish a uniform depression discrimination criterion and often overlook inter-individual differences in facial movements, which lead to a decrease in the accuracy of models. A closer examination of the clinical diagnostic criteria for depression reveals that individuals with depression exhibit a wide variety of symptoms. This suggests that even individuals with similar levels of depression may exhibit significantly different facial movements. To address this challenge, this study introduces a novel framework for depression assessment—Multiscale Video Adaptive Graph Network (MVAGN). By leveraging the information transmission capabilities of Graph Neural Network (GNN) to balance the relationship between individual differences and commonalities to achieve more accurate depression identification. MVAGN consists of three modules: the Multiscale Video Feature Extractor (MVFE), the Hybrid Feature Interaction Graph Module (HFIGM), and the Adaptive Hybrid Graph Network (AHGN). MVFE extracts multi-scale features from the video, HFIGM transforms these features into graph representations, and AHGN facilitates information transfer between nodes to reduce individual feature differences, ultimately providing a more precise estimation of depression severity. Experiments conducted on the AVEC2013 and AVEC2014 datasets demonstrate that MVAGN achieves state-of-the-art performance. This paper proposes that building a fuzzy system with a compact kernel and a loose periphery provides a viable path for recognizing depression.

Depression is a prevalent mental illness that significantly impacts the well-being of individuals and the development of society. The current diagnostic methods are largely subjective and time-consuming. Moreover, the existing machine learning-based depression recognition methods struggle to fully exploit the collaborative benefits between modalities, lack interpretability in their fusion processes, and perform inadequately in few-shot depression recognition tasks. To address these challenges, we propose a meta-fuzzy multimodal fusion network (MF$^{2}$-Net) for depression recognition. This innovative approach integrates physiological signals and behavioral data, employs multiple MLPs to learn the fuzzy measures of single base learners and complementary increments, then constructs all fuzzy measures, and finally achieves an interpretable decision-level fusion process through fuzzy integrals. Furthermore, we incorporate model-agnostic meta-learning for conducting few-shot domain-adaptive training, mitigating the issues related to high individual variability levels and the scarcity of multimodal depression data. Our method demonstrates exceptional classification performance in subject-independent experiments implemented on public datasets; offers a reliable solution for objectively, effectively, and conveniently recognizing depression; and has the potential to promote the clinical applications of rapid intelligent depression diagnosis.

Schizophrenia (SZ) is a severe mental disorder characterized by hallucinations, delusions, cognitive impairments, and social withdrawal. It leads to a series of brain abnormalities, particularly the deformation of the hippocampus and amygdala, which are highly associated with emotion, memory, and motivation. Most previous studies have used the hippocampal and amygdaloid volume, whereas surface-based morphometry reflects nuclear deformation more finely, but it is unclear the hippocampal and amygdaloid morphometry relates to schizophrenic pathology and its potential as a biomarker. In this study, we extracted individual multivariate morphometry statistics (MMS) of hippocampus and amygdala from MRI images and analyzed the morphometric differences between groups. After dictionary learning and max pooling, we obtain reduced dimensional features and use machine learning algorithms for individual diagnosis. The results showed that the hippocampus of the schizophrenia group was significantly atrophied bilaterally and the atrophied areas were symmetrical. Subregions of the amygdala are both atrophied and expanded, and in particular, the right amygdala shows a greater degree and extent of deformation. Using the random forest classifier, the accuracy of classification using hippocampal and amygdaloid morphometric features are 94.52% and 94.57%, respectively, and the accuracy of classification combining the two morphometric features reached 96.57%. Our study demonstrates the efficacy of MMS in identifying morphometric differences of the hippocampus and amygdala between healthy controls and schizophrenic, and these findings emphasize the potential of MMS as a reliable biomarker for the diagnosis of schizophrenia.

Federated learning (FL) has gained prominence in electroencephalogram (EEG)-based emotion recognition because of its ability to enable secure collaborative training without centralized data. However, traditional FL faces challenges due to model and data heterogeneity in smart healthcare settings. For example, medical institutions have varying computational resources, which creates a need for personalized local models. Moreover, EEG data from medical institutions typically face data heterogeneity issues stemming from limitations in participant availability, ethical constraints, and cultural differences among subjects, which can slow model convergence and degrade model performance. To address these challenges, we propose FedKDC, a novel FL framework that incorporates clustered knowledge distillation (CKD). This method introduces a consensus-based distributed learning mechanism to facilitate the clustering process. It then enhances the convergence speed through intraclass distillation and reduces the negative impact of heterogeneity through interclass distillation. Additionally, we introduce a DriftGuard mechanism to mitigate client drift, along with an entropy reducer to decrease the entropy of aggregated knowledge. The framework is validated on the SEED, SEED-IV, SEED-FRA, and SEED-GER datasets, demonstrating its effectiveness in scenarios where both the data and the models are heterogeneous. Experimental results show that FedKDC outperforms other FL frameworks in emotion recognition, achieving a maximum average accuracy of 85.2%, and in convergence efficiency, with faster and more stable convergence.

The development of affective computing and medical electronic technologies has led to the emergence of Artificial Intelligence (AI)-based methods for the early detection of depression. However, previous studies have often overlooked the necessity for the AI-assisted diagnosis system to be wearable and accessible in practical scenarios for depression recognition. In this work, we present an on-board executable multi-feature transfer-enhanced fusion model for our custom-designed wearable three-lead Electroencephalogram (EEG) sensor, based on EEG data collected from 73 depressed patients and 108 healthy controls. Experimental results show that the proposed model exhibits low-computational complexity (65.0 K parameters), promising Floating-Point Operations (FLOPs) performance (25.6 M), real-time processing (1.5 s/execution), and low power consumption (320.8 mW). Furthermore, it requires only 202.0 KB of Random Access Memory (RAM) and 279.6 KB of Read-Only Memory (ROM) when deployed on the EEG sensor. Despite its low computational and spatial complexity, the model achieves a notable classification accuracy of 95.2%, specificity of 94.0%, and sensitivity of 96.9% under independent test conditions. These results underscore the potential of deploying the model on the wearable three-lead EEG sensor for assisting in the diagnosis of depression.

Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.

Constrained decoding approaches aim to control the meaning or style of text generated by a Pre-trained Language Model (PLM) for various task-specific objectives at inference time. However, these methods often guide plausible continuations by greedily and explicitly selecting targets, which, while fulfilling the task requirements, may overlook the natural patterns of human language generation. In this work, we propose a novel decoding framework,Decider, which enables us to program high-level rules on how we might effectively complete tasks to control a PLM. Differing from previous works, our framework transforms the encouragement of concrete target words into the encouragement of all words that satisfy the high-level rules. Specifically,Decideris a dual system in which a PLM is equipped and controlled by a First-Order Logic (FOL) reasoner to express and evaluate the rules, along with a decision function that merges the outputs from both systems to guide the generation. Experiments on CommonGen and PersonaChat demonstrate thatDecidercan effectively follow given rules to guide a PLM in achieving generation tasks in a more human-like manner.

Heart sounds play a crucial role in assessing Coronary Artery Disease (CAD). The advancement of Artificial Intelligence (AI) technologies has given rise to Computer Audition (CA)-based methods for CAD detection. However, previous research has focused primarily on analyzing and modeling heart sound data, overlooking practical application scenarios. In this work, we design a pervasive heart sound collection device used for high-quality heart sound data acquisition. Moreover, we introduce an on-board executable lightweight network tailored for the designed portable device, referred to as TYKDModel. Further, heart sound data from 41 CAD patients and 22 non-CAD healthy controls are collected using the developed device. Experimental results show that the TYKDModel exhibits low-computational complexity, with 52.16 K parameters and 5.03 M Floating-Point Operations (FLOPs). When deployed on the board, it requires only 1.10 MB of Random Access Memory (RAM) and 236.27 KB of Read-Only Memory (ROM), and takes around 1.72 seconds to perform a classification. Despite the low computational and spatial complexity, the TYKDModel achieves a notable classification accuracy of 85.2%, specificity of 88.6%, and sensitivity of 82.8% on the board. These results indicate the promising potential of AI-assisted all-in-one integrated system for the diagnosis of heart sound-assisted CAD.

Depression detection via wearable Electroencephalogram (EEG) sensor-assisted diagnosis system demands computationally efficient models compatible with resource-constrained edge devices. Spiking Neural Networks (SNNs) offer inherent advantages for processing the spatio-temporal patterns of EEG through event-driven manner. In this study, we innovatively present LSNNet, a lightweight SNN model specifically designed for wearable EEG sensors. The model exhibits low computational complexity with 7.18 K parameters and 67.68 M Floating-Point Operations (FLOPs). It requires only 246.88 KB of Random Access Memory (RAM) and 57.33 KB of Read-Only Memory (ROM) for on-board execution, and has been validated on both the single-core STM32U535CET6 and the multi-core GAP8 microcontrollers. Despite its minimal computational and memory requirements, LSNNet achieves impressive performance metrics, with a classification accuracy of 89.2%, specificity of 92.4%, and sensitivity of 86.4% in independent tests conducted on EEG data collected from 73 depressed patients and 108 healthy controls using our three-lead EEG sensor. Especially, when running on the GAP8 microcontrollers, the LSNNet model has a low power consumption of 21.43 mW and a satisfactory inference time of 0.63 s while maintaining a classification accuracy of 87.5% (only with a reduction of 1.98%). These results underscore the potential of integrating wearable EEG sensors with the LSNNet model for depression detection in the Internet of Things (IoT) era.

Traditional clustering methods rely on pairwise affinity to divide samples into different subgroups. However, high-dimensional small-sample (HDLSS) data are affected by the concentration effects, rendering traditional pairwise metrics unable to accurately describe relationships between samples, leading to suboptimal clustering results. This article advances the proposition of employing high-order affinities to characterize multiple sample relationships as a strategic means to circumnavigate the concentration effects. We establish a nexus between different order affinities by constructing specialized decomposable high-order affinities, thereby formulating a uniform mathematical framework. Building upon this insight, a novel clustering method named uniform tensor clustering (UTC) is proposed, which learns a consensus low-dimensional embedding for clustering by the synergistic exploitation of multiple-order affinities. Extensive experiments on synthetic and real-world datasets demonstrate two findings: 1) high-order affinities are better suited for characterizing sample relationships in complex data and 2) reasonable use of different order affinities can enhance clustering effectiveness, especially in handling high-dimensional data.

Transfer learning is one of the popular methods to solve the problem of insufficient data in subject-specific electroencephalogram (EEG) recognition tasks. However, most existing approaches ignore the difference between subjects and transfer the same feature representations from source domain to different target domains, resulting in poor transfer performance. To address this issue, we propose a novel subject-specific EEG recognition method named deep multiview module adaption transfer (DMV-MAT) network. First, we design a universal deep multiview (DMV) network to generate different types of discriminative features from multiple perspectives, which improves the generalization performance by extensive feature sets. Second, module adaption transfer (MAT) is designed to evaluate each module by the feature distributions of source and target samples, which can generate an optimal weight sharing strategy for each target subject and promote the model to learn domain-invariant and domain-specific features simultaneously. We conduct extensive experiments in two EEG recognition tasks, i.e., motor imagery (MI) and seizure prediction, on four datasets. Experimental results demonstrate that the proposed method achieves promising performance compared with the state-of-the-art methods, indicating a feasible solution for subject-specific EEG recognition tasks. Implementation codes are available at https://github.com/YangLibuaa/DMV-MAT.

Granger causality (GC) effective connectivity (EC) calculated from electroencephalogram (EEG) signals has been widely used in mental disorder detection. However, the existing methods only take into account linear dynamics or nonlinear dynamics within a single sample, ignoring the nonlinear dynamics shared by the same class of subjects. In this article, a model combining graph neural networks (GNNs) and variational autoencoders (VAEs) is proposed to construct shared latent nonlinear EC from raw EEG signals for depression detection. Several convolution modules and fully connected layers are used in the graph encoding network to learn the embeddings of the connectivity connected by every two EEG channels. In the graph decoding network, a class-specific Gaussian mixture model (GMM) is introduced in the VAEs to model shared dynamics in EC of the same class of subjects, and the shared dynamics combine the encoded embeddings of the EC and the past time series to restore raw EEG signals. Through a node-to-edge encoding process and an edge-to-node decoding process, the shared latent nonlinear EC in EEG signals can ultimately be learned by gradually optimizing the model's loss function. The performance of the proposed method is verified on several open-accessed datasets. The excellent results prove that the proposed neural networks can learn more generalized nonlinear EC representations, and shared latent dynamics discovery can also help to identify depression better. The code is available at https://github.com/william-yuan2012/DSLNEC-tscausality.

Vigilance estimation is a critical task within the field of brain-computer interfaces, extensively applied in monitoring and optimizing user states during human-machine interaction using electroencephalography (EEG). However, most existing vigilance prediction frameworks are prone to spurious correlations stemming from inherent biases in collected data. These biases involve relevant but vigilance-independent information, which may lack robustness when applied to different data distributions, i.e., out-of-distribution (OOD) scenarios. The core idea of this study is to learn constraints that capture causal information from the input based on the assumed underlying data generating process. Leveraging the disentanglement and invariance principles behind the assumptions, we propose a constraint-driven causal representation learning (CCRL) to identify and separate spurious latent variables from biased training data for generalized vigilance estimation. The CCRL training process consists of two phases: self-supervised pretraining and constraint-driven causal information disentanglement. In the first phase, based on the masked autoencoder (MAE) architecture, unlabeled training data are used for reconstructing pretext tasks to capture the comprehensive and intrinsic contextual information from EEG data, which provides a powerful input for downstream disentanglement learning. In the second phase, we propose a novel disentanglement strategy to learn spurious-free latent representations causally related to the vigilance state driven by adversarial and invariance constraints. Comprehensive validation experiments conducted on two well-known public datasets demonstrate the effectiveness and superiority of the proposed framework. In general, this work has promising implications for addressing OOD challenges in vigilance estimation.

With social development, the demand for automatic sleep quality assessment in wearable devices is increasing, especially as sleep quality is closely related to the diagnosis of psychiatric disorders. However, existing automatic sleep stage classification models are mostly designed for unimodal signals, with an emphasis on increasing parameter scale and model depth. As a result, it is challenging to meet the requirements for both lightweight models and high accuracy in wearable device-based automatic sleep staging tasks. To address this problem, this study introduces a novel lightweight model for sleep staging, MultimodalSleepNet, which is based on multi-modal physiological signals. Specifically, the model is designed to capture the temporal dynamics of physiological signals and the spatial interactions between multimodal signals. Additionally, an inflationary convolution mechanism is incorporated to accelerate temporal feature extraction. We validate the model using the publicly available Sleep-EDF-Expanded dataset. Compared to similar studies, our model achieves outstanding performance, with accuracies of 93.1% and 90.2% in the three-stage and five-stage sleep recognition tasks, respectively. Notably, the three-stage classification results show an 11.9% improvement in modal fusion accuracy compared to unimodal signals, with an 8.9% improvement in multiclass F1 score and a 20.8% increase in Cohen’s kappa coefficient. In conclusion, our study offers a reference for the design of lightweight models for sleep staging and provides new insights into feature extraction and fusion of multimodal signals.

Previous studies have shown that the resting-state electroencephalogram (EEG) of depressed patients exhibits abnormal dynamic activation of large-scale networks in microstate analysis. However, the problem of heterogeneous results in microstate features as physiological markers for depressive disorders has not been addressed. An important factor contributing to this problem is that previous studies have overlooked the effects of eye-opening and eye-closing on resting-state EEG microstates in depression. To address this gap, the present study proposes a new microstate delineation method to accurately identify EEG microstates in both open-eye and closed-eye resting states, and further analyzes the differential performance of the depressed group and the control group in the two states. We validated our method on a dataset containing 64-channel EEG data (55 cases in the depressed group and 55 cases in the control group). The results showed that the classical seven microstate topographies were present in both open-eye and closed-eye conditions, which may explain why other studies have overlooked the impact of eye state on resting-state microstates in depression. In addition, the microstate characteristics of depression were differentially expressed in the open-eye and closed-eye conditions, with the depressed group showing more significant results in the closed-eye condition. Overall, our study demonstrates that eye state affects depression microstates and provides new insights to address the problem of heterogeneity in the results of EEG microstate studies of depression.

Depressive affective dysfunction could be manifested by dynamic reorganization processes of functional brain networks under emotional tasks. However, such emotional task-specific reorganizations remain not fully understood in terms of spatiotemporal organization of oscillation dynamics. The insufficiency of approaches for quantifying such dynamic reorganization limits the effective extraction of dynamic neuromarkers of depressive affective dysfunction. To address this gap, this study presents a neural oscillation-dependent microstate approach to quantify the dynamic reorganization process of functional networks at high temporal resolution. The approach was tested by analyzing a 64-channel electroencephalography (EEG) dataset with 110 participants (55 depressed patients and 55 normal controls) collected during emotional tasks with four affective polarities (positive, neutral, negative, and resting state). Our analyses revealed oscillation-dependent microstates that reflected abnormalities in the networks associated with external information processing and interoception in depression. Our analyses further suggest that such abnormalities may be caused by dysfunction in a limited number of brain regions, which then dynamically affects functional brain networks. These findings may reflect increased self-focus and deficits in the perception of external information in depression. In summary, our study further explores source-level evidence related to abnormalities in microstate features of depression based on the generalization of depression microstate research to a multi-band framework. Our study provides a direction for expanding the application of dynamic reorganization analyses of functional networks with high temporal resolution, and provides new support and insights into the neural mechanisms of affective dysfunction in depression.

Sleep is a daily activity essential for well-being, yet many modern individuals experience sleep problems, and prolonged poor sleep negatively impacts both physiological and psychological health. Aromatherapy, an emerging complementary alternative medicine, has shown promise as a sleep aid, but it lacks objective and reliable monitoring and control methods. To address this gap, we propose a wearable, portable sleep monitoring and aromatherapy system that utilizes single-channel electroencephalogram (EEG) signals from the prefrontal lobe for sleep detection and provides aromatherapy feedback based on the detected sleep state. Our system incorporates a lightweight model based on convolutional neural networks (CNN) and long short-term memory (LSTM) networks, enabling on-board execution and classification. Trained on the publicly available Sleep-EDF dataset, the model achieves a classification accuracy of 85.1% and a macro-F1 score of 79.5%. The combination of our developed EEG sensor and the proposed model presents a promising solution for effective sleep monitoring and intervention, aiming to enhance sleep quality.

Currently, treatment options for patients with treatment-resistant depression (TRD) are extremely limited. Although the rapid antidepressant effects of nitrous oxide have been preliminarily validated, the underlying neurophysiological mechanisms remain unclear. In this study, we intended to explore impact on brain modules induced by nitrous oxide through a randomized controlled study. A total of 44 patients with TRD were recruited. Subjects were randomly allocated to nitrous oxide-intervention group (1-hour inhalation of 50% nitrous oxide/50% oxygen) or placebo-control group (1-hour inhalation of 50% oxygen/50% air). The eye-closed resting state EEG signals were recorded. Our findings reveal that nitrous oxide treatment significantly altered the segregation of interconnected brain functional modules. The result of General Linear Model (repeated measurements) demonstrates superior antidepressant efficacy of nitrous oxide compared to placebo, as evidenced by reductions in both the participation coefficient and connector hub after treatment. Furthermore, changes in modular metrics moderately correlated with reductions in depressive symptoms. These findings offer valuable insights into mechanism underlying the treatment of TRD using nitrous oxide from a modular perspective.

In contemporary society, the issue of psychological health has become increasingly prominent, characterized by the diversification, complexity, and universality of mental disorders. Cognitive Behavioral Therapy (CBT), currently the most influential and clinically effective psychological treatment method with no side effects, has limited coverage and poor quality in most countries. In recent years, researches on the recognition and intervention of emotional disorders using large language models (LLMs) have been validated, providing new possibilities for psychological assistance therapy. However, are large language models truly possible to conduct cognitive behavioral therapy? Many concerns have been raised by mental health experts regarding the use of LLMs for therapy. Seeking to answer this question, we collected real CBT corpus from online video websites, designed and conducted a targeted automatic evaluation framework involving three aspects, namely the evaluation of emotion tendency of generated text, structured dialogue pattern and proactive inquiry ability. Considering limited CBT-related texts in a general chat LLM’s training corpus, we evaluated the CBT ability of the LLM after integrating a CBT knowledge base to explore the influence of introducing additional knowledge. Four LLM variants with exceptional performance are evaluated, and the experimental result shows the great potential of LLMs in psychological counseling realm, especially after combining with other technological means.

Computerized Cognitive Behavioral Therapy (CCBT) has been widely proven effective in alleviating negative emotions such as depression and anxiety. To address the lack of real-time feedback, we explored how to add appropriate interactions from the user experience perspective. By analyzing user context, we propose a transformation framework from CBT to interactive CCBT (iCCBT), which makes online interventions more effective. According to the framework, we designed and implemented an iCCBT platform for young adults. Through a 9-day controlled experiment on 28 college students, we conclude that interaction can effectively improve users’ knowledge retention, learning efficiency, satisfaction, and willingness to use. This study demonstrates methods and results for enhancing the interactivity of CCBT. The design thinking of the transformation framework can be extended to other online psychotherapy.

Cardiovascular diseases (CVDs) constitute the primary cause of human mortality globally in recent decades. To effectively detect CVDs, heart auscultation plays an important role in early diagnosis. With the development of artificial intelligence (AI), many studies have designed varying AI-assisted diagnosis systems helping people discriminate abnormal heart sounds. Yet, a robust system usually requires a noise-less input signal, which is critical as heart sounds are often affected by some unavoidable noise. Therefore, many heart sound classification models use filters or other methods to obtain the clean signals. However, these classic techniques are not adaptable enough to distinguish the meaningful murmurs and real noises. Thus, we propose a novel approach to transfer an audio source separation model to denoise the heart sound. In this paper, we test different denoisers on synthesis heart sound with additive white Gaussian noises. Our method performs well on the noise reduction metrics. Meanwhile, we evaluate the classification performance of each denoiser with some classifiers on the PhysioNet dataset. Experimental results demonstrate that our method can outperform other denoising techniques by achieving the highest unweighted average recall (UAR) at 95.7% with the smallest standard deviation. The results confirm that our method is robust and adaptable in improving audio's denoising.

Cardiovascular diseases (CVDs) stand as the primary reason of fatalities globally, especially in low- and middle-income countries. In recent years, with the leverage of computer audition technologies, the diagnosis of CVDs through heart sounds become a popular topic. Current models and techniques are trained, validated, and tested on the same dataset, which need to be retrained when encountering new data. To make the best use of sparse data, we propose a Prototypical Network framework with heuristic weight for heart sound recognition. After extracting two different features (Mel Spectrogram and Mel Frequency Cepstral Coefficients) and encoding the features, we calculate the distance between two categories (normal and abnormal), then, a heuristic weight is assigned to the distance that makes the blurred boundaries more distinct. By considering the subject independence, the Unweighted Average Recall (UAR) on the PhysioNet/CinC Challenge 2016 is 68.2 % and 67.7 % on two features, respectively. The capability of our model to work on different datasets is proved by a UAR of 66.4 %, which exceeds the baseline UAR of 58.6 % under a single model.

This study introduces a novel Supervised Info-enhanced Contrastive Learning framework for EEG based Emotion Recognition (SI-CLEER). SI-CLEER employs multi-granularity contrastive learning to create robust EEG contextual representations, potentially improving emotion recognition effectiveness. Unlike existing methods solely guided by classification loss, we propose a joint learning model combining self-supervised contrastive learning loss and supervised classification loss. This model optimizes both loss functions, capturing subtle EEG signal differences specific to emotion detection. Extensive experiments demonstrate SI-CLEER’s robustness and superior accuracy on the SEED dataset compared to state-of-the-art methods. Furthermore, we analyze electrode performance, highlighting the significance of central frontal and temporal brain region EEGs in emotion detection. This study offers an universally applicable approach with potential benefits for diverse EEG classification tasks.

Medical image segmentation is an important task in modern analysis of medical images. Current methods tend to extract either local features with convolutions or global features with Transformers. However, few of them are able to effectively fuse global and local features to facilitate segmentation. In this work, we propose a novel hybrid network that involves three main branches: the Multi-Layer Perception (MLP) branch, the Convolutional Neural Network (CNN) branch, and a Fusion branch. The MLP and CNN branches aim to learn global and local features, respectively. To fuse these, the fusion branch introduces a novel hierarchical fusion that performs multi-layered fusions that generate high-level representations to enhance segmentation. Our evaluation with two datasets shows strong performance of the proposed method compared to state-of-the-art baselines.

Phonocardiogram classification methods using deep neural networks have been widely applied to the early detection of cardiovascular diseases recently.Despite their excellent recognition rate, the sizeable computational complexity limits their further development.Nowadays, knowledge distillation (KD) is an established paradigm for model compression.While current research on multi-teacher KD has shown potential to impart more comprehensive knowledge to the student than single-teacher KD, this approach is not suitable for all scenarios.This paper proposes a novel KD strategy to realise an adaptive multi-teacher instruction mechanism.We design a teacher selection strategy called voting network to tell the contribution of different teachers on each distillation points, so that the student can choose the useful information and renounce the redundant one.An evaluation demonstrates that our method reaches excellent accuracy (92.8 %) while maintaining a low computational complexity (0.7 M).

Remote photoplethysmography (rPPG) measurement aims to estimate physiological signals by analyzing subtle skin color changes induced by heartbeats in facial videos. Existing methods primarily rely on the fundamental video frame features or vanilla facial ROI (region of interest) features. Recognizing the varying light absorption and reactions of different facial regions over time, we adopt a new perspective to conduct a more fine-grained exploration of the key clues present in different facial regions within each frame and across temporal frames. Concretely, we propose a novel clustering-driven remote physiological measurement framework called Cluster-Phys, which employs a facial ROI prototypical clustering module to adaptively cluster the representative facial ROI features as facial prototypes and then update facial prototypes with highly semantic correlated base ROI features. In this way, our approach can mine facial clues from a more compact and informative prototype level rather than the conventional video/ROI level. Furthermore, we also propose a spatial-temporal prototype interaction module to learn facial prototype correlation from both spatial (across prototypes) and temporal (within prototype) perspectives. Extensive experiments are conducted on both intra-dataset and cross-dataset tests. The results show that our Cluster-Phys achieves significant performance improvement with less computation consumption. The source code will be available at https://github.com/VUT-HFUT/ClusterPhys.

Unmanned aerial vehicles (UAVs) often require real-time and accurate control in military, rescue, and transportation applications. There is an urgent demand from operators for usability and interoperability. Human-machine collaboration enhancement and tiny machine learning aimed at edge computing hold the potential to greatly improve the operational efficiency of UAVs. In this paper, we propose a novel UAV control framework based on eye-tracking technology, i.e. Eye-Tracking-based UAV (ETUAV), which combines with lightweight object detection to assist the control of the UAV for attitude adjustment through the gaze information of the UAV operator. The eye-tracking-based UAV control method we propose involves real-time tracking of the operator’s gaze using our self-developed head-mounted eye tracker, combined with object detection to achieve automatic targeting. We also propose an incremental proportion integration differentiation (PID) control algorithm for adjusting the UAV’s attitude, which provides automatic and real-time UAV control. We develop a system prototype based on the DJI UAV and conduct performance benchmarks and comparisons. The experimental results indicate that the operational latency of our method is significantly less than manual operation and the built-in automatic tracking function in the DJI application. The “Aim Where You Look” UAV operation framework proposed in this article significantly streamlines the tasks for operators in time-critical scenarios.

Tensor spectral clustering (TSC) is an emerging approach that explores multi-wise similarities to boost learning. However, two key challenges have yet to be well addressed in the existing TSC methods: (1) The construction and storage of high-order affinity tensors to encode the multi-wise similarities are memory-intensive and hampers their applicability, and (2) they mostly employ a two-stage approach that integrates multiple affinity tensors of different orders to learn a consensus tensor spectral embedding, thus often leading to a suboptimal clustering result. To this end, this paper proposes a tensor spectral clustering network (TSC-Net) to achieve one-stage learning of a consensus tensor spectral embedding, while reducing the memory cost. TSC-Net employs a deep neural network that learns to map the input samples to the consensus tensor spectral embedding, guided by a TSC objective with multiple affinity tensors. It uses stochastic optimization to calculate a small part of the affinity tensors, thereby avoiding loading the whole affinity tensors for computation, thus significantly reducing the memory cost. Through using an ensemble of multiple affinity tensors, the TSC can dramatically improve clustering performance. Empirical studies on benchmark datasets demonstrate that TSC-Net outperforms the recent baseline methods.

In multi-view environment, it would yield missing observations due to the limitation of the observation process. The most current representation learning methods struggle to explore complete information by lacking either cross-generative via simply filling in missing view data, or solidative via inferring a consistent representation among the existing views. To address this problem, we propose a deep generative model to learn a complete generative latent representation, namely Complete Multi-view Variational Auto-Encoders (CMVAE), which models the generation of the multiple views from a complete latent variable represented by a mixture of Gaussian distributions. Thus, the missing view can be fully characterized by the latent variables and is resolved by estimating its posterior distribution. Accordingly, a novel variational lower bound is introduced to integrate view-invariant information into posterior inference to enhance the solidative of the learned latent representation. The intrinsic correlations between views are mined to seek cross-view generality, and information leading to missing views is fused by view weights to reach solidity. Benchmark experimental results in clustering, classification, and cross-view image generation tasks demonstrate the superiority of CMVAE, while time complexity and parameter sensitivity analyses illustrate the efficiency and robustness. Additionally, application to bioinformatics data exemplifies its practical significance.

Smartphones are widely used as portable data collectors for wearable and healthcare sensors that can passively collect data streams related to the environment, health status, and behaviors. Recent research shows that the collected data can be used to monitor not only the physical states but also the mental health of individuals. However, extracting the features of digital phenotypes that characterize major depressive disorder (MDD) is technically challenging and may raise significant privacy concerns. Addressing such challenges has become the focus of many researchers. This article provides a comprehensive analysis of several key issues related to ubiquitous sensing to aid in detecting MDD. Specifically, this article analyzes existing methodologies and feature extraction algorithms used to detect possible MDD through digital phenotyping from smartphone data. In particular, five types of features are summarized and explained, namely, location, movement, rhythm, sleep, and social and device usage. Finally, related limitations and challenges are discussed to provide paths for further research and engineering.

Although multimodal physiological data from the central and peripheral nervous systems can objectively respond to human emotional states, the individual differences caused by non-stationary and low signal-to-noise properties bring several challenges to cross-subject emotion recognition tasks. Many previous studies usually focused on learning high correlation information between different modalities, which easily leads to incomplete descriptions of different physiological signals and difficulties in aligning critical emotional information. To tackle these challenges, this paper proposes a novel multimodal emotion recognition model for improving the generalization performance to unseen target domain subjects, termed Completeness-induced Adaptative Broad Learning (CiABL). The proposed CiABL can gradely explore the completeness modality representation that encompasses both modality-relevant and modality-independent information, avoiding the loss of performance due to spurious correlations from different modalities. Subsequently, a well-designed weighted representation distribution alignment mechanism of CiABL can appropriately align the marginal and conditional distributions to reduce the influences of individual differences greatly. Extensive experiments on the SEED and SEED-FRA datasets demonstrate the effectiveness and generalization of the proposed CiABL, which outperforms current state-of-the-art methods. In addition, CiABL can precisely quantify the importance of global features to properly explain the modality contribution and averaged activation patterns of the brain under cross-subject emotion recognition tasks.

Emotion recognition in conversation (ERC) based on multiple modalities has attracted enormous attention. However, most research simply concatenated multimodal representations, generally neglecting the impact of cross-modal correspondences and uncertain factors, and leading to the cross-modal misalignment problems. Furthermore, recent methods only considered simple contextual features, commonly ignoring semantic clues and resulting in an insufficient capture of the semantic consistency. To address these limitations, we propose a novel multi-level alignment and cross-modal unified semantic graph refinement network (MA-CMU-SGRNet) for ERC task. Specifically, a multi-level alignment (MA) is first designed to bridge the gap between acoustic and lexical modalities, which can effectively contrast both the instance-level and prototype-level relationships, separating the multimodal features in the latent space. Second, a cross-modal uncertainty-aware unification (CMU) is adopted to generate a unified representation in joint space considering the ambiguity of emotion. Finally, a dual-encoding semantic graph refinement network (SGRNet) is investigated, which includes a syntactic encoder to aggregate information from near neighbors and a semantic encoder to focus on useful semantically close neighbors. Extensive experiments on three multimodal public datasets show the effectiveness of our proposed method compared with the state-of-the-art methods, indicating its potential application in conversational emotion recognition. Implementation codes can be available athttps://github.com/zxiaohen/MA-CMU-SGRNet.

Stress has been identified as one of major causes of health issues. To detect the stress levels with higher accuracy, fusion of multimodal physiological signals is a promising technique. However, there is an asynchrony between physiological signals observed from different perspectives. Exploring the temporal alignment relationship between modalities is helpful to improve the quality of multimodal fusion. This paper proposes an end-to-end multimodal stress detection model based on Bidirectional Cross- and Self-modal Attention (BCSA) mechanism. Specifically, we first construct different feature extractors based on the characteristics of Blood Volume Pulse (BVP) and Electrodermal Activity (EDA) to complete automated temporal feature extraction. Secondly, cross-modal attention is used to seek the alignment relationship between the two modalities and fully fuse cross-modal information. The self-modal attention is used to attenuate noise and redundant information, highlight important information and obtain salient stress representations. Finally, the stress representations of the two modalities are processed separately, and the mean square error (MSE) is used to narrow the gap between them. Experimental results on the UBFC-Phys dataset and WESAD dataset show that the proposed model can effectively improve the accuracy of stress recognition, and outperforms several state-of-the-art methods.

With great pride and enthusiasm, we present the inaugural edition of IEEE Transactions on Computational Social Systems (TCSS) for 2024. Reflecting on the year gone by, 2023 stands as a hallmark of academic excellence and prolific output, wherein our journal has successfully disseminated a substantial volume of scholarly work—301 articles encompassing approximately 3600 pages, distributed across six distinct issues.

Welcome to the second issue of IEEE Transactions on Computational Social Systems (TCSS) of 2024. This issue showcases an impressive array of 104 regular papers alongside our Special Issue on Big Data and Computational Social Intelligence for Guaranteed Financial Security, highlighting cutting-edge research aimed at harnessing big data and computational techniques to fortify financial security amidst the digital finance evolution. With a focus on addressing the intricate challenges of financial big data, enhancing the efficacy of artificial intelligence, and covering critical topics from data mining to digital currencies, this issue underscores the vital role of cross-disciplinary efforts in mitigating financial security risks.

This edition of the publication includes a robust collection of 104 regular papers and features a Special Issue on Knowledge- Infused Learning for Computational Social Systems. This special issue delves into the sophisticated integration of advanced technologies and knowledge-based methodologies within the analysis of computational social systems. Spanning 12 articles, the issue addresses a wide spectrum of topics, from big data management to refining machine learning models with domain-specific insights. It encompasses areas such as energy management in sensor networks, acoustic analysis of heartbeats, detection of fraudulent activities in online ratings, and the management of rumors on social networks, exemplifying the significant role that knowledge-infused learning plays in enhancing technological applications and fostering innovation in social computational systems.

Micro-action is an imperceptible non-verbal behaviour characterised by low-intensity movement. It offers insights into the feelings and intentions of individuals and is important for human-oriented applications such as emotion recognition and psychological assessment. However, the identification, differentiation, and understanding of micro-actions pose challenges due to the imperceptible and inaccessible nature of these subtle human behaviors in everyday life. In this study, we innovatively collect a new micro-action dataset designated as Micro-action-52 (MA-52), and propose a benchmark named micro-action network (MANet) for micro-action recognition (MAR) task. Uniquely, MA-52 provides the whole-body perspective including gestures, upper- and lower-limb movements, attempting to reveal comprehensive micro-action cues. In detail, MA-52 contains 52 micro-action categories along with seven body part labels, and encompasses a full array of realistic and natural micro-actions, accounting for 205 participants and 22,422 video instances collated from the psychological interviews. Based on the proposed dataset, we assess MANet and other nine prevalent action recognition methods. MANet incorporates squeeze-and-excitation (SE) and temporal shift module (TSM) into the ResNet architecture for modeling the spatiotemporal characteristics of micro-actions. Then a joint-embedding loss is designed for semantic matching between video and action labels; the loss is used to better distinguish between visually similar yet distinct micro-action categories. The extended application in emotion recognition has demonstrated one of the important values of our proposed dataset and method. In the future, further exploration of human behaviour, emotion, and psychological assessment will be conducted in depth. The dataset and source code are released at https://github.com/VUT-HFUT/Micro-Action.

Eye-tracking technology is extensively utilized in affective computing research, enabling the investigation of emotional responses through the analysis of eye movements. Integration of eye-tracking with other modalities, allows for the collection of multimodal data, leading to a more comprehensive understanding of emotions and their relationship with physiological responses. This paper presents a novel head-mounted eye-tracking system for multimodal data acquisition with a completely redesigned structure and improved performance. We propose a novel method for pupil-fitting with high efficiency and robustness based on deep learning and RANSAC, which gets better performance of pupil segmentation when it is partially occluded, and build a 3D model to obtain gaze points. Existing eye trackers for multi-modal synchronous data collection either have limited device support or suffer from significant synchronization delays. Our proposed hard real-time synchronization mechanism implements microsecond level latency with low cost, which facilitates multimodal analysis for affective computing research. The uniquely designed exterior effectively reduces facial occlusion, making it more comfortable for the wearer while facilitating the capture of facial expressions.

Modeling correlations between multimodal physiological signals [e.g., canonical correlation analysis (CCA)] for emotion recognition has attracted much attention. However, existing studies rarely consider the neural nature of emotional responses within physiological signals. Furthermore, during fusion space construction, the CCA method maximizes only the correlations between different modalities and neglects the discriminative information of different emotional states. Most importantly, temporal mismatches between different neural activities are often ignored; therefore, the theoretical assumptions that multimodal data should be aligned in time and space before fusion are not fulfilled. To address these issues, we propose a discriminative correlation fusion method coupled with a temporal alignment mechanism for multimodal physiological signals. We first use neural signal analysis techniques to construct neural representations of the central nervous system (CNS) and autonomic nervous system (ANS). respectively. Then, emotion class labels are introduced in CCA to obtain more discriminative fusion representations from multimodal neural responses, and the temporal alignment between the CNS and ANS is jointly optimized with a fusion procedure that applies the Bayesian algorithm. The experimental results demonstrate that our method significantly improves the emotion recognition performance. Additionally, we show that this fusion method can model the underlying mechanisms in human nervous systems during emotional responses, and our results are consistent with prior findings. This study may guide a new approach for exploring human cognitive function based on physiological signals at different time scales and promote the development of computational intelligence and harmonious human-computer interactions.

The tracking control of redundant manipulators plays a crucial role in robotics research and generally requires accurate knowledge of models of redundant manipulators. When the model information of a redundant manipulator is unknown, the trajectory-tracking control with model-based methods may fail to complete a given task. To this end, this article proposes a data-driven neural dynamics-based model predictive control (NDMPC) algorithm, which consists of a model predictive control (MPC) scheme, a neural dynamics (ND) solver, and a discrete-time Jacobian matrix (DTJM) updating law. With the help of the DTJM updating law, the future output of the model-unknown redundant manipulator is predicted, and the MPC scheme for trajectory tracking is constructed. The ND solver is designed to solve the MPC scheme to generate control input driving the redundant manipulator. The convergence of the proposed data-driven NDMPC algorithm is proven via theoretical analyses, and its feasibility and superiority are demonstrated via simulations and experiments on a redundant manipulator. Under the drive of the proposed algorithm, the redundant manipulator successfully carries out the trajectory-tracking task without the need for its kinematics model.

Functional connectivity (FC) networks, built from analyses of resting-state magnetic resonance imaging (rs-fMRI), serve as efficacious biomarkers for identifying Autism Spectrum Disorders (ASD) patients. Given the neurobiological heterogeneity across individuals and the unique presentation of ASD symptoms, the fusion of individualized information into diagnosis becomes essential. However, this aspect is overlooked in most methods. Furthermore, the existing methods typically focus on studying direct pairwise connections between brain ROIs, while disregarding interactions between indirectly connected neighbors. To overcome above challenges, we build common FC and individualized FC by tangent pearson embedding (TP) and common orthogonal basis extraction (COBE) respectively, and present a novel multiview brain transformer (MBT) aimed at effectively fusing common and indivinformation of subjects. MBT is mainly constructed by transformer layers with diffusion kernel (DK), fusion quality-inspired weighting module (FQW), similarity loss and orthonormal clustering fusion readout module (OCFRead). DK transformer can incorporate higher-order random walk methods to capture wider interactions among indirectly connected brain regions. FQW promotes adaptive fusion of features between views, and similarity loss and OCFRead are placed on the last layer to accomplish the ultimate integration of information. In our method, TP, DK and FQW modules all help to model wider connectivity in the brain that make up for the shortcomings of traditional methods. We conducted experiments on the public ABIDE dataset based on AAL and CC200 respectively. Our framework has shown promising results, outperforming state-of-the-art methods on both templates. This suggests its potential as a valuable approach for clinical ASD diagnosis.

Text content analysis for depression detection using machine learning techniques has become a prominent area of research. However, previous studies focused mainly on analyzing the textual content, neglecting the fundamental factors driving text generation. Consequently, existing models face the challenge of poor generalization to out-of-domain data as they struggle to capture the crucial features of depression. To address this, we propose a novel computational perspective of "stimulus-response patterns" that brings us closer to the essence of clinical diagnosis of depression. Adopting this computational perspective allows us to conceptually unify diverse datasets and generalize this perspective to common datasets in the field. We introduce the Stimulus-Response Patterns-aware Network (SRP-Net) as an exemplary approach within this computational perspective. To assess the performance of the SRP-Net, we constructed a multi-stimulus dataset and conducted experimental evaluations, demonstrating its exceptional cross-stimulus generalizability. Furthermore, we demonstrated the promising performance of SPR-Net in real medical scenarios and conducted an interpretability analysis of the stimulus-response patterns. Our research investigates the critical role of stimulus-response patterns in enhancing the generalizability of text-based depression detection models, which can potentially facilitate data-driven depression detection to approach the diagnostic accuracy of psychiatrists.

Recently, psychophysiological computing has received considerable attention. Due to easy acquisition at a distance and less conscious initiation, gait-based emotion recognition is considered as a valuable research branch in the field of psychophysiological computing. However, most existing methods rarely explore the spatio-temporal context of gait, which limits the ability to capture the higher-order relationship between emotion and gait. In this paper, we utilize a range of research, including psychophysiological computing and artificial intelligence, to propose an integrated emotion perception framework called EPIC, which can find novel joint topology and generate thousands of synthetic gaits by spatio-temporal interaction context. First, we analyze the joint coupling among non-adjacent joints by calculating Phase Lag Index (PLI), which can discover the latent connection among body joints. Second, to synthesize more sophisticated and accurate gait sequences, we explore the effect of spatio-temporal constraints, and propose a new loss function that utilizes the Dynamic Time Warping (DTW) algorithm and pseudo-velocity curve to constrain the output of Gated Recurrent Units (GRU). Finally, Spatial Temporal Graph Convolution Networks (ST-GCN) is used to classify emotions using the generation and the real data. Experimental results demonstrate our approach achieves the accuracy of 89.66%, and outperforms the state-of-the-art methods on Emotion-Gait dataset.

Ubiquitous sensing has been widely applied in smart healthcare, providing an opportunity for intelligent heart sound auscultation. However, smart devices contain sensitive information, raising user privacy concerns. To this end, federated learning (FL) has been adopted as an effective solution, enabling decentralised learning without data sharing, thus preserving data privacy in the Internet of Health Things (IoHT). Nevertheless, traditional FL requires the same architectural models to be trained across local clients and global servers, leading to a lack of model heterogeneity and client personalisation. For medical institutions with private data clients, this study proposes Fed-MStacking, a heterogeneous FL framework that incorporates a stacking ensemble learning strategy to support clients in building their own models. The secondary objective of this study is to address scenarios involving local clients with data characterised by inconsistent labelling. Specifically, the local client contains only one case type, and the data cannot be shared within or outside the institution. To train a global multi-class classifier, we aggregate missing class information from all clients at each institution and build meta-data, which then participates in FL training via a meta-learner. We apply the proposed framework to a multi-institutional heart sound database. The experiments utilise random forests (RFs), feedforward neural networks (FNNs), and convolutional neural networks (CNNs) as base classifiers. The results show that the heterogeneous stacking of local models performs better compared to homogeneous stacking.

Depression is a prevalent mental disorder that affects a significant portion of the global population. Despite recent advancements in EEG-based depression recognition models rooted in machine learning and deep learning approaches, many lack comprehensive consideration of depression's pathogenesis, leading to limited neuroscientific interpretability. To address these issues, we propose a hemisphere asymmetry network (HEMAsNet) inspired by the brain for depression recognition from EEG signals. HEMAsNet employs a combination of multi-scale Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) blocks to extract temporal features from both hemispheres of the brain. Moreover, the model introduces a unique 'Callosum-like' block, inspired by the corpus callosum's pivotal role in facilitating inter-hemispheric information transfer within the brain. This block enhances information exchange between hemispheres, potentially improving depression recognition accuracy. To validate the performance of HEMAsNet, we first confirmed the asymmetric features of frontal lobe EEG in the MODMA dataset. Subsequently, our method achieved a depression recognition accuracy of 0.8067, indicating its effectiveness in increasing classification performance. Furthermore, we conducted a comprehensive investigation from spatial and frequency perspectives, demonstrating HEMAsNet's innovation in explaining model decisions. The advantages of HEMAsNet lie in its ability to achieve more accurate and interpretable recognition of depression through the simulation of physiological processes, integration of spatial information, and incorporation of the Callosum-like block.

Brain anatomical age is an effective feature to assess the status of the brain, such as atypical development and aging. Although some deep learning models have been developed for estimating infant brain age, the performance of these models was unsatisfactory because few of them considered the developmental characteristics of brain anatomy during the perinatal period-the most rapid and complex developmental stage across the lifespan. The present study proposed an attention-based hemispheric relation inference network (HRINet) that takes advantage of the nature of brain structural lateralization during early development. This model captures the inter-hemispheric relationship using a graph attention mechanism and transmits lateralization information as features to describe the interactive development between bilateral hemispheres. The HRINet was used to estimate the brain age of 531 preterm and full-term neonates from the Developing Human Connectome Project (dHCP) database based on two metrics (mean curvature and sulcal depth) characterizing the folding morphology of the cortex. Our results showed that the HRINet outperformed other benchmark models in fitting the perinatal brain age, with mean absolute error of 0.53 and determination coefficient of 0.89. We also verified the generalizability of the HRINet on an extra independent dataset collected from the Gansu Provincial Maternity and Child-care Hospital. Furthermore, by applying the best-performing model to an independent dataset consisting of 47 scans of preterm infants at term-equivalent age, we showed that the predicted age was significantly lower than the chronological age, suggesting a delayed development of premature brains. Our results demonstrate the effectiveness and generalizability of the HRINet in estimating infant brain age, providing promising clinical applications for assessing neonatal brain maturity.

Depression is one of the most common mental illnesses, but few of the currently proposed in-depth models based on social media data take into account both temporal and spatial information in the data for the detection of depression. In this paper, we present an efficient, low-covariance multimodal integrated spatio-temporal converter framework called DepMSTAT, which aims to detect depression using acoustic and visual features in social media data. The framework consists of four modules: a data preprocessing module, a token generation module, a Spatial-Temporal Attentional Transformer (STAT) module, and a depression classifier module. To efficiently capture spatial and temporal correlations in multimodal social media depression data, a plug-and-play STAT module is proposed. The module is capable of extracting unimodal spatio-temporal features and fusing unimodal information, playing a key role in the analysis of acoustic and visual features in social media data. Through extensive experiments on a depression database (D-Vlog), the method in this paper shows high accuracy (71.53%) in depression detection, achieving a performance that exceeds most models. This work provides a scaffold for studies based on multimodal data that assists in the detection of depression.

Conventional functional connectivity network (FCN) based on resting-state fMRI (rs-fMRI) can only reflect the relationship between pairwise brain regions. Thus, the hyper-connectivity network (HCN) has been widely used to reveal high-order interactions among multiple brain regions. However, existing HCN models are essentially spatial HCN, which reflect the spatial relevance of multiple brain regions, but ignore the temporal correlation among multiple time points. Furthermore, the majority of HCN construction and learning frameworks are limited to using a single template, while the multi-template carries richer information. To address these issues, we first employ multiple templates to parcellate the rs-fMRI into different brain regions. Then, based on the multi-template data, we propose a spatio-temporal weighted HCN (STW-HCN) to capture more comprehensive high-order temporal and spatial properties of brain activity. Next, a novel deep fusion model of multi-template called spatio-temporal weighted multi-hypergraph convolutional network (STW-MHGCN) is proposed to fuse the STW-HCN of multiple templates, which extracts the deep interrelation information between different templates. Finally, we evaluate our method on the ADNI-2 and ABIDE-I datasets for mild cognitive impairment (MCI) and autism spectrum disorder (ASD) analysis. Experimental results demonstrate that the proposed method is superior to the state-of-the-art approaches in MCI and ASD classification, and the abnormal spatio-temporal hyper-edges discovered by our method have significant significance for the brain abnormalities analysis of MCI and ASD.

Current cross-modal retrieval methods heavily rely on accurate semantic labels or sample similarity measurements, and need to search for the nearest samples among all samples in the huge search space, severely limiting the application in stratifying large-scale and high-dimensional multimodal data. To tackle with the issues, this paper proposes an unsupervised cross-modal retrieval method to bypass the semanticwise supervision and samplewise similarity from a standpoint of featurewise matching, named by unsupervised dual hashing coding (UDC). It jointly learns the dual hashing codes on semantic tagging and sample content through factorizing a feature matching potential, which is allowed to bridge the semantic and heterogeneous gaps among different modalities simultaneously through maintaining the inter-modality-consistent semantic information and cross-modality-correlated sample content. In this way, each sample is uniquely coded by a head code on semanticwise tags, and tail codes on samplewise content. The dual coding design makes it very efficient for sample retrieval, in which the query sample only need to search for the retrieved ones with the same semantic tag, greatly narrowing down the search space. The proposed model avoids the calculation of massive sample-wise similarity and works with dual hashing coding scheme, which achieves a twofold efficiency enhancement for analyzing the large-scale and high-dimensional multimodal data. Extensive experiments have been conducted to demonstrate that it achieved superiority on computational time and retrieval performance.

Mounting evidence shows that Alzheimer's disease (AD) manifests the dysfunction of the brain network much earlier before the onset of clinical symptoms, making its early diagnosis possible. Current brain network analyses treat high-dimensional network data as a regular matrix or vector, which destroys the essential network topology, thereby seriously affecting diagnosis accuracy. In this context, harmonic waves provide a solid theoretical background for exploring brain network topology. However, the harmonic waves are originally intended to discover neurological disease propagation patterns in the brain, which makes it difficult to accommodate brain disease diagnosis with high heterogeneity. To address this challenge, this article proposes a network manifold harmonic discriminant analysis (MHDA) method for accurately detecting AD. Each brain network is regarded as an instance drawn on a Stiefel manifold. Every instance is represented by a set of orthonormal eigenvectors (i.e., harmonic waves) derived from its Laplacian matrix, which fully respects the topological structure of the brain network. An MHDA method within the Stiefel space is proposed to identify the group-dependent common harmonic waves, which can be used as group-specific references for downstream analyses. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method in stratifying cognitively normal (CN) controls, mild cognitive impairment (MCI), and AD.

At present, there have been many studies on the methods of using the deep learning regression model to assess depression level based on behavioral signals (facial expression, speech, and language); however, the research on the assessment method of anxiety level using deep learning is absent. In this article, pupil-wave, a physiological signal collected by Human Computer Interaction (HCI) that can directly represent the emotional state, is developed to assess the level of depression and anxiety for the first time. In order to distinguish between different depression and anxiety levels, we use the HCI method to induce the participants’ emotional experience through three virtual reality (VR) emotional scenes of joyful, sad, and calm, and construct two differential pupil-waves of joyful and sad with the calm pupil-wave as the baseline. Correspondingly, a dual-channel fusion depression and anxiety level assessment model is constructed using the improved multi-scale convolution module and our proposed width-channel attention module for one-dimensional signal processing. The test results show that the MAE/RMSE of the depression and anxiety level assessment method proposed in this article is 3.05/4.11 and 2.49/1.85, respectively, which has better assessment performance than other related research methods. This study provides an automatic assessment technique based on human computer interaction and virtual reality for mental health physical examination.

Gaze following aims to predict where a person is looking in a scene. Existing methods tend to prioritize traditional 2D RGB visual cues or require burdensome prior knowledge and extra expensive datasets annotated in 3D coordinate systems to train specialized modules to enhance scene modeling. In this work, we introduce a novel framework deployed on a simple ResNet backbone, which exclusively uses image and depth maps to mimic human visual preferences and realize 3D-like depth perception. We first leverage depth maps to formulate spatial-based proximity information regarding the objects with the target person. This process sharpens the focus of the gaze cone on the specific region of interest pertaining to the target while diminishing the impact of surrounding distractions. To capture the diverse dependence of scene context on the saliency gaze cone, we then introduce a learnable grid-level regularized attention that anticipates coarse-grained regions of interest, thereby refining the mapping of the saliency feature to pixel-level heatmaps. This allows our model to better account for individual differences when predicting others’ gaze locations. Finally, we employ the KL-divergence loss to super the grid-level regularized attention, which combines the gaze direction, heatmap regression, and in/out classification losses, providing comprehensive supervision for model optimization. Experimental results on two publicly available datasets demonstrate the comparable performance of our model with less help of modal information. Quantitative visualization results further validate the interpretability of our method. The source code will be available at https://github.com/VUT-HFUT/DepthMatters .

Depression is an illness that involves emotional and mental health. Currently, depression detection through interviews is the most popular way. With the advancement of natural language processing and sentiment analysis, automated interview-based depression detection is strongly supported. However, current multimodal depression detection models fail to adequately capture the fine-grained features of depressive behaviors, making it difficult for the models to accurately characterize the subtle changes in depressive symptoms. To address this problem, we propose a Multi Fine-Grained Fusion Network (MFFNet). The core idea of this model is to extract and fuse the information of different scale feature pairs through a Multi-Scale Fastformer (MSfastformer), and then use the Recurrent Pyramid Model to integrate the features of different resolutions, promoting the interaction of multi-level information. Through the interaction of multi-scale and multi-resolution features, it aims to explore richer feature representations. To validate the effectiveness of our proposed MFFNet model, we conduct experiments on two depression interview datasets. The experimental results show that the MFFNet model performs better in depression detection compared to other benchmark multimodal models.

This paper focuses on contrastive learning for gait-based emotion recognition. The existing contrastive learning approaches are rarely suitable for learning skeleton-based gait representations, which suffer from limited gait diversity and inconsistent semantics. In this paper, we propose a Cross-coordinate contrastive learning framework utilizing Ambiguity samples for self-supervised Gait-based Emotion representation (CAGE). First, we propose ambiguity transform to push positive samples into ambiguous semantic space. By learning similarities between ambiguity samples and positive samples, our model can learn higher-level semantics of the gait sequences and maintain semantic diversity. Second, to encourage learning the semantic invariance, we uniquely propose cross-coordinate contrastive learning between the Cartesian coordinate and the Spherical coordinate, which brings rich supervisory signals to learn the intrinsic semantic consistency information. Exhaustive experiments show that CAGE improves existing self-supervised methods by 5%–10% accuracy, and it achieves comparable or even superior performance to supervised methods.

Functional magnetic resonance imaging (fMRI) plays a vital role in brain science as it measures and maps brain activity through the analysis of blood flow changes, offering valuable insights into cognitive functions and neural processes. However, due to the intricacy and dynamism of brain activity, conventional approaches failed to accurately predict the blood oxygen level-dependent (BOLD) time series in fMRI data. To tackle this issue, we proposed an end-to-end adversarial U-network (AUN) to verify the predictability of existing BOLD signals in primary cortex (i.e., primary visual, primary motor and primary sensory) and higher cortex(i.e., dorsolateral prefrontal and posterior cingulate). The model combined the U-network architecture and adversarial strategy to ensure that the predicted results capture both the intricate nonlinear details and the overall distribution characteristics. We performed the experiment using the Human Connectome Project (HCP) database. The results demonstrated the predictability of both primary and higher cortex, with primary cortex showing higher predictability. Additionally, the AUN performed better than other popular methods. We also found the improvement in dynamic functional connectivity (dFC) metrics through accurate prediction. The above results confirm the feasibility of predicting BOLD signals and their potential application in clinical settings.1

Sublingual vein features including swelling, varicose and cyanosis are essential for the symptoms differentiation and treatment selection in Traditional Chinese Medicine (TCM) tongue diagnosis, especially reflecting the state of human blood circulation. However, automatic and accurate extraction of sublingual vein features remains a great challenge, limited by both the lack of datasets for sublingual images and the influence of noise from non-tongue and non-sublingual vein components. In this paper, we propose a novel tongue features extraction method based on segmenting the sublingual vein instead of the whole tongue bottom, in which a sublingual vein segmentation framework based on a Polyp-PVT network is developed to eliminate the noise from the surrounding part of the sublingual vein. Meanwhile, we first adopt a transformer-based method such as Swin-Transformer network to extract sublingual vein features by virtue of the awesome capability of the transformer network. In addition, we construct a large dataset including 4018 sublingual vein images for the segmentation and classification of sublingual veins. Experimental results have shown that the tongue feature extraction method combined with a sublingual vein segmentation can greatly outperform the existing tongue feature extracting methods.

Many studies attempt to explore the underlying mechanisms of depression and distinguish between depression patients and normal controls (NC) using electroencephalography (EEG) techniques. With the advancement of deep learning methods, an increasing number of studies aim to design Computer-Aided Diagnosis (CAD) systems for mild depression (MD) to achieve early identification. However, few studies construct models based on abnormal brain topological structures specific to MD patients. In this study, we investigate the abnormal brain topological structures of individuals with MD based on EEG data obtained during an emotional face paradigm. Functional connectivity analysis reveals a higher proportion of inter-hemispheric connections in the MD group compared to intra-hemispheric connections. Additionally, intra-hemispheric connections are primarily observed within the frontal and parietal lobes of both groups. Hierarchical clustering analysis results indicate impairments in the frontal and parietal lobes in the MD group compared to the NC group. Based on these findings, we propose a novel feature called "cross-brain feature" and introduce a multi-cross-brain attention fusion mechanism to integrate information between brain regions. We train and test our models using 5-fold cross-validation. The results demonstrate that the classification model based on abnormal brain topological structures achieves the highest performance among the three state-of-the-art algorithms, with an accuracy of 80.1%, an area under the ROC curve (AUC) of 80%, and a sensitivity (SEN) of 86.3%. These findings suggest that combining abnormal brain topological structures derived from functional connectivity matrices with deep learning techniques can provide an effective objective approach to the early detection of depression.

Depression is a prevalent mental disorder that poses significant risks to human health and social stability. Current methods for diagnosing depression heavily rely on patient descriptions and psychiatrist observations, which are susceptible to interference from subjective factors and carry the risk of misdiagnosis and missed diagnosis. Therefore, it is crucial to develop an objective method for recognizing depression based on objective criteria. Recently, combining EEG signals with deep learning techniques for depression recognition has become a popular research topic. However, existing EEG-based depression recognition methods are poorly interpretable, making it challenging to explain the neural mechanisms of depression disorders. Consequently, we propose an explainable framework for depression recognition from EEG signals based on a GCN. In this method, a hybrid module of 1DCNN, LSTM and GCN is utilized to extract features from EEG signals, which can effectively capture spatiotemporal correlations between different brain regions. The EEG subgraph construction module explores the differences in crucial connectivity patterns of the brain between different groups, enhancing the interpretability of our model. The experimental results on the MODMA dataset show that our model outperforms the baseline model in all metrics, thus verifying the validity of the proposed model. Additionally, compared with existing explainable algorithms, our method consistently yielded nearly identical experimental results, demonstrating its ability to capture the correlation between depression and neuroscience, and has good interpretability.

Event-related potentials (ERP) within the Electroencephalogram (EEG), in particular, are electric signals induced by stimuli that can reflect specific cognitive activities of the brain. Therefore, ERP can be used as an objective biomarker to distinguish patients with depression from healthy individuals. However, the analysis and classification of single-trial ERP are difficult due to the high trial-to-trial variability and the low signal-to-noise ratio (SNR). Therefore, how to improve the SNR and the single-trial classification accuracy has received much attention. In this study, we proposed a spatiotemporal linear discriminant analysis (STLDA) method for single-trial ERP-based depression recognition, which can obtain optimal temporal and spatial filters for ERP signals to enhance their SNR significantly and preserve the spatiotemporal characteristics of the signals for depression recognition using the minimum distance to mean (MDM) classification strategy. Experimental results on two public datasets showed that our method achieved higher classification accuracy in comparison with some baseline methods. Further, the depression recognition performance of our method is almost equal to the traditional trial-averaged strategy.

This study presents a novel Supervised Information Enhanced Contrastive Learning Algorithm for Parkinson’s Disease Detection (SI-CLAPD) based on Electroencephalography (EEG). SI-CLAPD performs contrastive learning in a multi-granularity manner on enhanced contextual views to achieve robust contextual representations for EEG, thus could contribute to improving the effectiveness of PD detection. Unlike existing methods for constructing PD detection models guided solely by classification loss, we propose a joint learning model that combines self-supervised contrastive learning with supervised classification learning. This model is optimized using both contrastive loss and classification loss, allowing it to capture subtle differences between EEG signals and representations that are specific to PD detection. Through extensive experimental evaluations, we demonstrate that SI-CLAPD achieves robust and high accuracy in PD detection tasks on three benchmark datasets. To the best of our knowledge, this study represents the first effort in validating the effectiveness of contrastive learning for the detection of PD. Besides, within the realm of contrastive learning research in EEG, it also represents the first endeavor to fuse supervised learning with self-supervised contrastive learning for EEG classification. This investigation unveils an universally applicable approach to EEG signal processing, with the potential to confer advantages to a multitude of EEG classification tasks.

Mental fatigue is a prevalent issue in contemporary society and can negatively affect physical performance and concentration, increasing the likelihood of adverse consequences due to inattention during productive activities. Therefore, it becomes increasingly important to address and eliminate fatigue within a specific period of time. Aromatherapy, as a form of Complementary Alternative Medicine (CAM), is a non-invasive, cost-effective, and efficient method to combat fatigue. Previous studies have assessed the effects of specific aromatherapy oils using scales, but there is a lack of objective and reliable physiological indicators to prove the effectiveness of aromatherapy. Hence, this paper seeks to establish a model illustrating the effects of aromatic essential oil gases on the human body. A multimodal physiological fatigue signal acquisition system that integrates aromatherapy feedback was designed. In addition, an experimental paradigm was developed to explore the potential of aromatherapy in mitigating mental fatigue. Electroencephalogram (EEG) and Electrocardiogram (ECG) signals were collected, allowing for the analysis of time-frequency domain features in EEG and ECG signals, as well as Heart Rate Variability (HRV) features in ECG signals. Our findings indicate that specific aromatic gases demonstrate effectiveness in reducing mental fatigue. Furthermore, we employed the Support Vector Machine (SVM) algorithm to classify the state of human mental fatigue. Based on the classification results, the release of aromatic gas was controlled to provide targeted aromatic feedback. This innovative approach offers a promising avenue for objectively assessing and addressing mental fatigue through aromatherapy interventions.

Mental health has long been studied, and many AI-based approaches have been proposed for diagnosis and adjunctive therapy. The emergence of Pre-trained Large Language Models (LLMs) has had a profound impact on various fields, but the potential of using ChatGPT for cognitive behavior therapy is largely unexplored. Therefore, there is an urgent need to build and design a virtual interactive framework for assisted diagnosis/treatment. In this paper, we present a virtual interaction framework based on LLMs that allows participants to engage in a dialogue with a virtual character, analyse mental health issues through augmented LLMs, and make suggestions during the dialogue to alleviate the psychological problems they are currently facing. Based on this framework, we develop a use case for the application of ChatGPT in the field of emotional disorders. Specifically, we use data from question-and-answer dialogues in real-life scenarios to populate the current exploration of ChatGPT’s potential for depression and anxiety detection. The case study shows the great potential of ChatGPT in the analysis of depression and anxiety tests. The feasibility of a virtual interaction framework based on LLMs has been preliminarily demonstrated.

Deep learning methods have been widely adopted in the field of computer-aided EEG diagnosis, one of the major topics to be investigated is the input format of EEG data. Related researches have reported the application of Fast Fourier Transform (FFT) to topology-preserving multi-spectral images generation on three frequency bands (i.e. theta, alpha and beta) jointed to preserve the spatial information. In our work, we proposed a new approach of using Empirical Mode Decomposition (EMD) instead of FFT algorithm to generate topology-preserving multi-spectral images on intrinsic mode functions (IMFs) jointed and only on the single IMF. Meanwhile, images generated on three frequency bands jointed and the single frequency band were also used for comparison. We then applied two convolutional neural network (CNN) structures to distinguish depression patients and normal subjects from the topology-preserving multi-spectral images. As a result, both convolutional networks obtain better accuracies on three IMFs jointed (about 5% better, ≈ 75% vs. ≈ 70%) than three frequency bands jointed. Analysis based on the single frequency band indicates that alpha band performs the best, with an accuracy of more than 78%, and among all classification results, the best classification accuracy obtained is 80.23% on IMF2. The results are encouraging, despite the limited size of our cohort, the use of EMD and our findings cast a new light on application of deep learning method to EEG-based depression recognition.

The cardiovascular diseases (CVDs) cause tremendous deaths yearly. The Mel-spectrogram is widely used as a tool to analyse the heart sound, which facilitate a cheap and efficient diagnosis of CVDs. Nevertheless, the amplitude and frequency responses of the Mel filter banks remain constant, limiting its function to frequency selection. We propose an adaptive Melspectrogram end-to-end neural network (AMNet) for a better characterisation and classification of heart sound in the work. The core of the adaptive Mel-spectrograms (AMel) lies in an adaptive Mel filter banks whose frequency characteristics remain the same as the original Mel-spectrogram (OMel) and amplitude is learnt by the backropagation algorithm. The AMNet learns the raw audio representation directly and outputs the classification results. It reaches 43.5% Unweighted Average Recall (UAR) and surpasses the model with the OMel and the baseline by 6% UAR. It is demonstrated that the AMel characterises the heart sound more effectively.

Translating mental health recognition from clinical research into real-world application requires extensive data, yet existing emotion datasets are impoverished in terms of daily mental health monitoring, especially when aiming for self-reported anxiety and depression recognition. We introduce the Japanese Daily Speech Dataset (JDSD), a large in-the-wild daily speech emotion dataset consisting of 20,827 speech samples from 342 speakers and 54 hours of total duration. The data is annotated on the Depression and Anxiety Mood Scale (DAMS) – 9 self-reported emotions to evaluate mood state including "vigorous", "gloomy", "concerned", "happy", "unpleasant", "anxious", "cheerful", "depressed", and "worried". Our dataset possesses emotional states, activity, and time diversity, making it useful for training models to track daily emotional states for healthcare purposes. We partition our corpus and provide a multi-task benchmark across nine emotions, demonstrating that mental health states can be predicted reliably from self-reports with a Concordance Correlation Coefficient value of .547 on average. We hope that JDSD will become a valuable resource to further the development of daily emotional healthcare tracking.

Stuttering is a complicated language disorder. The most common form of stuttering is developmental stuttering, which begins in childhood. Early monitoring and intervention are essential for the treatment of children with stuttering. Automatic speech recognition technology has shown its great potential for non-fluent disorder identification, whereas the previous work has not considered the privacy of users’ data. To this end, we propose federated intelligent terminals for automatic monitoring of stuttering speech in different contexts. Experimental results demonstrate that the proposed federated intelligent terminals model can analyze symptoms of stammering speech by taking personal privacy protection into account. Furthermore, the study has explored that the Shapley value approach in the federated learning setting has comparable performance to data-centralised learning.

The accuracy and availability are the most critical and challenging problems for major depressive disorder (MDD) diagnosis. Limited receptive field and inaccurate visual interpretation always weaken the clinical application of deep learning-based depression recognition model. Thus, we propose a visually interpretable depression monitoring model termed Transformer and Convolutional with slot-attention (TC-slot) to assess depression from facial images. Specifically, this approach stands upon the intersection of convolution and transformer, combines self-attention mechanism and deep convolution, and uses a well-designed stem structure to explore the global and local relationships. Moreover, in TC-slot, a classifier built on slot-attention mechanism directly involved in the decision-making process further localizes salient regions of facial depression patterns and provides precise and meaningful explanations. The results indicate that the proposed approach effectively improves the classification and recognition performance compared with other state-of-the-art approaches, with guaranteed favorable visual interpretability, providing clinical insights into the assessment of the assessing depression.

Human-Computer Interaction (HCI) and Machine Learning (ML) technologies have potential for the behavioral screening of autistic children but how to design a tool and analyse behavior reliably is challenging. Based on psychophysiological computation, this paper proposes an interactive behavior perception analytical model for autism screening. We presented the multi-scenario reactive behavior paradigms that designed based on the atypical characteristics of autistic children. We recorded the eye movement data and facial data of 91 participants, and performed multi-modal feature extraction, used machine learning to train classification model. We conducted comparative experiments, and the experimental results verified the advantages of multi-scenario paradigms and multi-modal feature groups, which indicates that our analysis methods and screening models are effective and reliable and have real research significance.

Abstract In recent years, mindfulness meditation has become increasingly popular as a way to relieve negative emotions. Although many studies have shown the advantages of the most immersive virtual reality (VR) technologies to support mindfulness meditation, few have summarized a standardized process for developing a VR tool to aid mindfulness meditation. We propose an interaction design model for VR mindfulness meditation using imagery‐based transformation and positive feedback to help users quickly soothe their negative emotions. Based on the traditional mindfulness meditation model, we propose the three detailed transformational steps to build (1) an interaction guidance process, (2) an interaction experience context, and (3) an interaction feedback mechanism. We build two scenarios based on them. Our experimental results suggest that our design can effectively alleviate a user's anxiety and provide emotional relief. Our insights on the transformation model of VR mindfulness meditation can be applied in related research, providing an effective reference model for VR mindfulness meditation systems, while providing new ideas for non‐pharmacological interventions in psychotherapy.

A brain–computer interface (BCI) enables a user to communicate directly with a computer using only the central nervous system. An affective BCI (aBCI) monitors and/or regulates the emotional state of the brain, which could facilitate human cognition, communication, decision-making, and health. The last decade has witnessed rapid progress in aBCI research and applications, but there does not exist a comprehensive and up-to-date tutorial on aBCIs. This tutorial fills the gap. It introduces first the basic concepts of BCIs and then, in detail, the individual components in a closed-loop aBCI system, including signal acquisition, signal processing, feature extraction, emotion recognition, and brain stimulation. Next, it describes three representative applications of aBCIs, i.e., cognitive workload recognition, fatigue estimation, and depression diagnosis and treatment. Several challenges and opportunities in aBCI research and applications, including brain signal acquisition, emotion labeling, diversity and size of aBCI datasets, algorithm comparison, negative transfer in emotion recognition, and privacy protection and security of aBCIs, are also explained.

It was observed that accuracy of the Subject-Dependent emotion recognition model was much higher than that of the Subject-Independent model in the field of electroencephalogram (EEG) based affective computing. This phenomenon is mainly caused by the individual difference of EEG, which is the key issue to be solved for the application of emotion recognition. In this work, 14 subjects from the SEED were selected for individual difference analysis. Through individual aggregation features evaluation, sample space visualization, and correlation analysis, we proposed four quantification indicators to analyze individual difference phenomenon. Finally, we presented the Personal-Zscore (PZ) feature processing method, and it was found that the data set processed with PZ method could represent emotion better than the original data set, and the conventional model with the PZ method was more robust. The accuracies of emotion recognition models trained with PZ processing have been improved to some extent, which showed that the PZ method could effectively eliminate the individual aggregation of feature space and improve the emotional representation ability of data sets. Hence, our findings may provide a new insight into the foundation for universal implementation of EEG-based application, and the Personal-Zscore feature processing method is of great significance for the development of effective emotion recognition system.

Emotional conflict control is impaired in major depression disorders (MDDs) and affects decision-making with further consequent social interactions dysfunction. However, neural correlates of conflict monitoring processes being modulated by different affective distractor stimuli are not clear in MDDs. In this article, we investigated abnormal neural basis of conflict monitoring processes in MDD patients by applying dynamic causal modeling (DCM) technique on electroencephalography (EEG). The results indicated that MDD patients showed lower N2 amplitudes regardless of stimulus conditions, and reduced activation within ACC region for incongruent stimuli, relative to healthy controls. Especially, MDDs had more negative N2 amplitudes to happy incongruent trials than happy congruent trials. Source localization analyses revealed that MDD patients had significantly enhanced left inferior temporal gyrus (ITG) activation, which is involved in written words processing. Further DCM analysis provided abnormal neural correlates through greater backward connections (fusiform→ITG, amygdala→ITG) on happy incongruent trials than happy congruent trials in MDD group. These findings indicate that only sad words induce significantly greater interference effects to positive target faces in MDD patients, which may be associated with ITG activity dysfunction. The findings may share new insights into the neural mechanisms of emotional conflict processing in MDDs.

Major depressive disorder (MDD) may be driven by dysfunction in intrinsic dynamic properties of the brain, and EEG microstate is a promising method for analyzing brain dynamics. However, the alterations in EEG microstate is still not entirely clear, and its ability for MDDs detection is worth probing. Moreover, the mechanism behind the neural networks contributing to microstates remains poorly understood in MDDs. Therefore, we applied microstate analysis and Topographic Electrophysiological State Source-imaging (TESS) on EEG data of 27 MDDs and 28 healthy controls (HCs). Compared to HCs, MDDs had apparent increase in microstate C and decrease in microstate D. Furthermore, TESS results showed that the underlying network of microstate C in MDDs overlapped with the anterior cingulate cortex and left insula gyrus, whereas main source of microstate D was in the orbital part of inferior frontal gyrus. The reduced transition probability from C to D in MDDs may reveal an imbalance between the networks of microstates. The microstate parameters as features reached good performance in identifying MDD (89.09% accuracy, 92.86% sensitivity, 85.19% specificity), indicating their potential as biomarkers of depression pathology. Collectively, these results highlight alteration of brain activity patterns and provide new insights into abnormal EEG dynamics in MDDs.

The detection of mild depression is conducive to the early intervention and treatment of depression. This study explored the fusion of electroencephalography (EEG) and pupil area signals to build an effective and convenient mild depression recognition model. We proposed Mutual Information Based Fusion Model (MIBFM), which innovatively used pupil area signals to select EEG electrodes based on mutual information. Then we extracted features from EEG and pupil area signals in different bands, and fused bimodal features using the denoising autoencoder. Experimental results showed that MIBFM could obtain the highest accuracy of 87.03%. And MIBFM exhibited better performance than other existing methods. Our findings validate the effectiveness of the use of pupil area as signals, which makes eye movement signals can be easily obtained using high resolution camera, and the EEG electrode selection scheme based on mutual information is also proved to be an applicable solution for data dimension reduction and multimodal complementary information screening. This study casts a new light for mild depression recognition using multimodal data of EEG and pupil area signals, and provides a theoretical basis for the development of portable and universal application systems.

Effective estimation of brain network connectivity enables better unraveling of the extraordinary complexity interactions of brain regions and helps in auxiliary diagnosis of psychiatric disorders. Considering different modalities can provide comprehensive characterizations of brain connectivity, we propose the message-passing-based nonlinear network fusion (MP-NNF) algorithm to estimate multimodal brain network connectivity. In the proposed method, the initial functional and structural networks were computed from fMRI and DTI separately. Then, we update every unimodal network iteratively, making it more similar to the others in every iteration, and finally converge to one unified network. The estimated brain connectivities integrate complementary information from multiple modalities while preserving their original structure, by adding the strong connectivities present in unimodal brain networks and eliminating the weak connectivities. The effectiveness of the method was evaluated by applying the learned brain connectivity for the classification of major depressive disorder (MDD). Specifically, 82.18% classification accuracy was achieved even with the simple feature selection and classification pipeline, which significantly outperforms the competing methods. Exploration of brain connectivity contributed to MDD identification suggests that the proposed method not only improves the classification performance but also was sensitive to critical disease-related neuroimaging biomarkers.

OBJECTIVE: Depression is accompanied by abnormalities in large-scale functional brain networks. This paper combined static and dynamic methods to analyze the abnormal topology and changes of functional connectivity network (FCN) of depression. METHODS: We collected resting-state EEG recordings from 27 depressed subjects and 28 normal subjects, then obtained 68 regions of interests (ROIs) by source localization. We took ROIs as the nodes and correlations as the edges to build FCNs and analyzed static network based on graph theory. We used a sliding window method followed by k-means clustering, states analyses and trend analysis of network metrics over time to study dynamic connectivity. RESULTS: The clustering coefficient (CC) and local efficiency in depression were increased, the characteristic path length and global efficiency were decreased, and local metrics had different manifestations in different resting state networks (RSNs); Depression had reduced connectivity in most RSNs, but increased connectivity in the default mode network, and there was a decoupling phenomenon between different RSNs; Depressed patients spent more time in sparsely connected states, their FCN's flexibility was less than normal subjects; The trend of CC over time was opposite between two groups. Most metrics in normal showed a relatively stronger correlation with time. SIGNIFICANCE: Our research may provide a deeper understanding of neurophysiological mechanisms of depression and new biomarkers for clinical diagnosis of depression.

Greetings and welcome to the third issue of IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS (TCSS) for 2023. The authors are delighted to share some exciting news with our esteemed readership.

Welcome to the fourth issue of IEEE Transactions on Computational Social Systems (TCSS) in 2023. We are pleased to share some significant developments in our journal. In late June, Clarivate released the updated Impact Factor for all journals indexed by the esteemed Web of Science database. We are delighted to announce that according to the Journal Citation Reports, the 2022 Journal Impact Factor of IEEE TCSS has been determined as 5. Furthermore, we are proud to inform you that IEEE TCSS has achieved a remarkable position in the JCR Category of computer science, cybernetics, ranking in Q1. Thank you once again for your tremendous efforts and unwavering support, which have contributed to the continued success of IEEE TCSS.

Welcome to the first issue of IEEE Transactions on Computational Social Systems (TCSS) of 2023. The past 2022 was again a very productive year, in which we have published 159 articles with about 1850 pages in six issues. We also received much great and exciting news.

Welcome to the second issue of the IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS (TCSS) of 2023. According to the latest update of CiteScoreTracker from Elsevier Scopus released on February 5, 2023, the CitesSore of TCSS has reached a historical high of 9.6. Many thanks to all for your great effort and support.

Greetings and welcome to the fifth issue of IEEE Transactions on Computational Social Systems (TCSS) for 2023. This edition presents a collection of 55 diverse regular articles that illuminate various facets of the interaction between computer technology and society.

Welcome to the concluding issue of IEEE Transactions on Computational Social Systems (TCSS) for the year 2023. We would like to seize this opportunity to extend our heartfelt appreciation and congratulations to all for your exceptional dedication and unwavering support. We eagerly anticipate further collaboration to enhance the publication quality and expedite the review process of TCSS in the upcoming year 2024.

Major depressive disorder (MDD) is one of the most common and severe mental illnesses, posing a huge burden on society and families. Recently, some multimodal methods have been proposed to learn a multimodal embedding for MDD detection and achieved promising performance. However, these methods ignore the heterogeneity/homogeneity among various modalities. Besides, earlier attempts ignore interclass separability and intraclass compactness. Inspired by the above observations, we propose a graph neural network (GNN)-based multimodal fusion strategy named modal-shared modal-specific GNN, which investigates the heterogeneity/homogeneity among various psychophysiological modalities as well as explores the potential relationship between subjects. Specifically, we develop a modal-shared and modal-specific GNN architecture to extract the inter/intramodal characteristics. Furthermore, a reconstruction network is employed to ensure fidelity within the individual modality. Moreover, we impose an attention mechanism on various embeddings to obtain a multimodal compact representation for the subsequent MDD detection task. We conduct extensive experiments on two public depression datasets and the favorable results demonstrate the effectiveness of the proposed algorithm.

Facial expression recognition (FER) in the wild is a challenging task for affective computing in human–machine interaction fields. However, most of the existing methods fail to learn the most prominent regions of facial images by simple cross-entropy loss due to the imbalance problem commonly existing in FER datasets, which limits the robustness and interpretability of the model. In addition, these methods only capture local features of original images with multisize shallow convolution and ignore facial texture characteristics, leading to a suboptimal recognition performance. To address these issues, in this article, we propose a novel FER network, named the attention-rectified and texture-enhanced cross-attention transformer feature fusion network (AR-TE-CATFFNet). Specifically, an attention-rectified convolution block is first designed to assist multiple convolution heads to focus on the critical areas of human faces and improve the model generalization. Second, we investigate a texture enhancement block to capture texture features through local binary pattern and gray-level co-occurrence matrix, which solves the limitation of insufficient texture information. Finally, a cross-attention transformer feature fusion block is employed to deeply integrate red, green, blue (RGB) features and texture features globally, which is beneficial to boost the accuracy of recognition. Competitive experimental results on three public datasets validate the efficacy of the proposed method, indicating that our proposed method achieves superior classification performance of 89.50% on real-world affective faces database (RAF-DB) dataset, 65.66% on AffectNet dataset, and 74.84% on FER2013 dataset against the existing methods.

Since brain network organization is essentially governed by the harmonic waves derived from the Eigen-system of the underlying Laplacian matrix, discovering the harmonic-based alterations provides a new window to understand the pathogenic mechanism of Alzheimer's disease (AD) in a unified reference space. However, current reference (common harmonic waves) estimation studies over the individual harmonic waves are often sensitive to outliers, which are obtained by averaging the heterogenous individual brain networks. To address this challenge, we propose a novel manifold learning approach to identify a set of outlier-immunized common harmonic waves. The backbone of our framework is calculating the geometric median of all individual harmonic waves on the Stiefel manifold, instead of Fréchet mean, thus improving the robustness of learned common harmonic waves to the outliers. A manifold optimization scheme with theoretically guaranteed convergence is tailored to solve our method. The experimental results on synthetic data and real data demonstrate that the common harmonic waves learned by our approach are not only more robust to the outliers than the state-of-the-art methods, but also provide a putative imaging biomarker to predict the early stage of AD.

Depression is a serious and common psychiatric disease characterized by emotional and cognitive dysfunction. In addition, the rates of clinical diagnosis and treatment for depression are low. Therefore, the accurate recognition of depression is important for its effective treatment. Electroencephalogram (EEG) signals, which can objectively reflect the inner states of human brains, are regarded as promising physiological tools that can enable effective and efficient clinical depression diagnosis and recognition. However, one of the challenges regarding EEG-based depression recognition involves sufficiently optimizing the spatial information derived from the multichannel space of EEG signals. Consequently, we propose an adaptive channel fusion method via improved focal loss (FL) functions for depression recognition based on EEG signals to effectively address this challenge. In this method, we propose two improved FL functions that can enhance the separability of hard examples by upweighting their losses as optimization objectives and can optimize the channel weights by a proposed adaptive channel fusion framework. The experimental results obtained on two EEG datasets show that the developed channel fusion method can achieve improved classification performance. The learned channel weights include the individual characteristics of each EEG epoch, which can effectively optimize the spatial information of each EEG epoch via the channel fusion method. In addition, the proposed method performs better than the state-of-the-art channel fusion methods.

Numerous studies have shown that accurate analysis of neurological disorders contributes to the early diagnosis of brain disorders and provides a window to diagnose psychiatric disorders due to brain atrophy. The emergence of geometric deep learning approaches provides a new way to characterize geometric variations on brain networks. However, brain network data suffer from high heterogeneity and noise. Consequently, geometric deep learning methods struggle to identify discriminative and clinically meaningful representations from complex brain networks, resulting in poor diagnostic accuracy. Hence, the primary challenge in the diagnosis of brain diseases is to enhance the identification of discriminative features. To this end, this paper presents a dual-attention deep manifold harmonic discrimination (DA-DMHD) method for early diagnosis of neurodegenerative diseases. Here, a low-dimensional manifold projection is first learned to comprehensively exploit the geometric features of the brain network. Further, attention blocks with discrimination are proposed to learn a representation, which facilitates learning of group-dependent discriminant matrices to guide downstream analysis of group-specific references. Our proposed DA-DMHD model is evaluated on two independent datasets, ADNI and ADHD-200. Experimental results demonstrate that the model can tackle the hard-to-capture challenge of heterogeneous brain network topological differences and obtain excellent classifying performance in both accuracy and robustness compared with several existing state-of-the-art methods.

Depression is a heterogeneous syndrome with certain individual differences among subjects. Exploring a feature selection method that can effectively mine the commonness intra-groups and the differences inter-groups in depression recognition is therefore of great significance. This study proposed a new clustering-fusion feature selection method. Hierarchical clustering (HC) algorithm was used to capture the heterogeneity distribution of subjects. Average and similarity network fusion (SNF) algorithms were adopted to characterize the brain network atlas of different populations. Differences analysis was also utilized to obtain the features with discriminant performance. Experiments showed that compared with traditional feature selection methods, HCSNF method yielded the optimal classification results of depression recognition in both sensor and source layers of electroencephalography (EEG) data. Especially in the beta band of EEG data at sensor layer, the classification performance was improved by more than 6%. Moreover, the long-distance connections between parietal-occipital lobe and other brain regions not only have high discriminative power, but also significantly correlate with depressive symptoms, indicating the important role of these features in depression recognition. Therefore, this study may provide methodological guidance for the discovery of reproducible electrophysiological biomarkers and new insights into common neuropathological mechanisms of heterogeneous depression diseases.

We focus on unsupervised representation learning for skeleton based action recognition. Existing unsupervised approaches usually learn action representations by motion prediction but they lack the ability to fully learn inherent semantic similarity. In this paper, we propose a novel framework named Prototypical Contrast and Reverse Prediction (PCRP) to address this challenge. Different from plain motion prediction, PCRP performs reverse motion prediction based on encoder-decoder structure to extract more discriminative temporal pattern, and derives action prototypes by clustering to explore the inherent action similarity within the action encoding. Specifically, we regard action prototypes as latent variables and formulate PCRP as an expectation-maximization (EM) task. PCRP iteratively runs (1) E-step as to determine the distribution of action prototypes by clustering action encoding from the encoder while estimating concentration around prototypes, and (2) M-step as optimizing the model by minimizing the proposed ProtoMAE loss, which helps simultaneously pull the action encoding closer to its assigned prototype by contrastive learning and perform reverse motion prediction task. Besides, the sorting can also serve as a temporal task similar as reverse prediction in the proposed framework. Extensive experiments on N-UCLA, NTU 60, and NTU 120 dataset present that PCRP outperforms main stream unsupervised methods and even achieves superior performance over many supervised methods. The codes are available at:https://github.com/LZUSIAT/PCRP.

Electroencephalogram (EEG) equipped with high time resolution that distracted by incoherent brain sources which including ocular artifacts (OAs) generating by blinks is of great significance. Therefore, these OAs got corrected before extracting information from EEG is indispensable. Improved Moment Matching (IMM) is a high-speed denoising algorithm suitable for removing OA in multi-channel EEG, which is an improvement of the moment matching method used to remove stripe noise in hyperspectral images. On foundation of this, this paper proposes an optimization algorithm for IMM based on Gaussian decay function (IMM_G). In the first place, the construction of the reference signal is optimized via utilizing a Gaussian decay function, thereby preventing the signal distortion caused by the filter. And then, a method for judging the to-be-processed interval is proposed to realizes the individualized processing of different channels. As a result, through quantitative comparison experiments with simulated data and real data from the UAIS laboratory, it was identified that IMM_G showed less time complexity, significantly enhanced arithmetic speed and denoising consequence while the detail retention ability of the original method for the non-blinking region of multi-channel EEG data is maintained. Hence, this method could be extensively used in High-density EEG (hdEEG) OAs removal scene.

Suicide causes serious harm to individuals, families and society, and becomes a social problem of widespread concern. Therefore, it is necessary to find and intervene individuals at risk of suicide as soon as possible. In recent years, social media data has successfully been leveraged for suicide risk detection. However, for fine-grained suicide risk detection, the existing models ignore the deviation between the predicted results and the real results when making wrong predictions, and do not pay attention to the semantic information contained in the labels. This paper proposes a deep learning model based on Label-Text Correlation and Deviation Punishment (LTC-DP). While learning the semantic relation adequately between the text and the corresponding label, the model can give different punishment adaptively according to the deviation degrees between the predicted results and the real result. The experimental results show that compared with the baseline model, the proposed model has better performance in fine-grained suicide risk detection. In addition, we release a fine-grained suicide risk detection data set based on Weibo, the data set is available at https://github.com/cxyazy/FGCSD-main.

Knowledge graphs (KGs) are widely used in various natural language processing applications. In order to expand the coverage of a KG, KG completion has attracted extensive attention. The commonly used embedding methods based on a large amount of training data can play an important role in this work. However, with few of triples, the performance of these methods will be greatly reduced. The completion of this kind of few-shot task is more challenging. In this work, we propose a method of data enhancement to increase the data quantity and solve the problem of sample shortage. Specifically, we first observe that the representation vectors of the relation in a KG are approximately subordinate to Gaussian distribution. Then we construct a Gaussian distribution for the relation of each triple in few-shot task according to the distributions of its similar relations in background graph. Further, we sample from the Gaussian distribution of each triple to expand the training data. Finally, we use an adaptive attentional network model FAAN proposed by Sheng et al. as the baseline model. Experimental results on two public datasets NELL-One and Wiki-One show that the proposed method achieves better performance.

In the Doppler biological radar-based applications of noncontact measurement of vital signs, effectively extracting heartbeat information from weak thoracic mechanical motion is an important problem to be solved. This study is aimed at extracting heartbeat signal via the technology combined with Short Time Fourier Transform (STFT), Singular Value Decomposition (SVD) and Adaptive Noise Canceller (ANC) from radar recording. The simulated data and the data collected by Doppler radar biosensor realized in laboratory are employed to validate the proposed method. The results show that the proposed method has the ability of detection for the heart rate and heart rate variability indexes in rest state, it has certain advantages in time-consuming and detection accuracy. Therefore, the current method provides another way to process vital sign signals recorded by Doppler radar.

Neural circuit dysfunction underlies the biological mechanisms of major depressive disorder (MDD). However, little is known about how the brain’s dynamic connectomes differentiate between depressed patients and normal controls. As a result, we collected resting-state Electroencephalography from 16 MDD patients and 16 controls using 128-electrode geodesic sensor net. Static and dynamic network metrics were later applied to explore the abnormal topological structure of MDD patients and identify them from normal controls using traditional machine learning algorithms with feature selection methods. Results showed that the MDD tend to have a more randomized formation both in static and dynamic network. We also found that the combined static-dynamic feature set usually outperformed others with a highest accuracy of 79.25% under delta band. Lower frequency band (delta, theta) showed relatively better outcomes compared to higher frequency band (alpha, beta). It also indicate the role of functional segregation features as a potential biomarker for depression. In conclusion, neuropathological mechanism of depression may be more objectively quantified and evaluated from the perspective of combining static and dynamic network.

Mental fatigue of the brain will cause the weakening of the psychological and physiological functions of the human body, which increases the risk of mistakes and accidents. This paper designs a mental fatigue detection and mitigation system based on human physiological signals and classical music. Through the fatigue assessment and mitigation experiment on 20 healthy subjects, it is verified that the system is able to judge the mental fatigue state of the subjects and play classical music to mitigate fatigue via acquiring and analyzing (alpha+theta)/beta of electroencephalogram (EEG) and heart rate variability (HRV) of ballistocardiogram (BCG). The designed system realizes signal acquisition and fatigue analysis only through using portable device, this makes it have certain application value in office building, hospital, aircraft driving and other scenes.

In recent years, depression recognition using physiological signals has achieved certain progress, but mild depression recognition is still in its infancy. Early detection can prevent the development of depression, and combining multiple modalities for analysis has been proved effective in the domain of mental disorders detection. Electroencephalogram (EEG) and eye movements (EM) are widely used to identify depression. However, the problem of using physiological signals to detect mental illness is that the generalization ability of the model is not strong, which is caused by individual differences. In view of the above problems, this paper proposes a hybrid fusion model based on deep belief network (DBN) and secondary classifier, called HFMBDSC, which first uses unsupervised DBN to fuse EEG features (linear, nonlinear features and network features) at the feature level. DBN transforms EEG features into another form to mitigate the effects of individual differences in EEG, and obtains DBN features that can more comprehensively represent EEG information. In the next step of decision level fusion, DBN features and EM features jointly make the final decision using the three classifiers that perform best when using single modality. The results show that DBN can improve the classification accuracy and effectively reduce the influence of individual differences in EEG features through visual analysis of feature space. The classification performance of HFMBDSC is significantly improved compared to the traditional single modality results. The highest accuracy rate of 89.54% is obtained under 10-fold cross-validation. These results suggest that mild depression recognition based on HFMBDSC is promising.

A plethora of great successes has been achieved by the existing convolutional neural networks (CNN) for respiratory sound classification. Nevertheless, simultaneously capturing both the local and global features can never be an easy task due to the limitation of a CNN’s structure. In this contribution, we propose a novel glance-and-gaze network to address the aforementioned issue. The glance block aims to learn global information, while the gaze block is responsible for learning local patterns and suppressing the noises that attenuates the final performance. In the proposed method, both the global and local information can be extracted. Moreover, the spectral and temporal representations can be learnt via a feature fusion module. Experimental results on the largest public respiratory sound database demonstrate that the proposed model outperforms the state-of-the-art methods.

Math Word Problems (MWP) aims to automatically solve mathematical questions given in texts. Previous studies tend to design complex models to capture additional information in the original text so as to enable the model to gain more comprehensive features. In this paper, we turn our attention in the opposite direction, and work on how to discard redundant features containing spurious correlations for MWP. To this end, we design an Expression Syntax Information Bottleneck method for MWP (called ESIB) based on variational information bottleneck, which extracts essential features of the expression syntax tree while filtering latent-specific redundancy containing syntax-irrelevant features. The key idea of ESIB is to encourage multiple models to predict the same expression syntax tree for different problem representations of the same problem by mutual learning so as to capture consistent information of expression syntax tree and discard latent-specific redundancy. To improve the generalization ability of the model and generate more diverse expressions, we design a self-distillation loss to encourage the model to rely more on the expression syntax information in the latent space. Experimental results on two large-scale benchmarks show that our model not only achieves state-of-the-art results but also generates more diverse solutions.

Alzheimer's disease (AD) has a strong genetic predisposition. However, its risk genes remain incompletely identified. We developed an Alzheimer's brain gene network-based approach to predict AD-associated genes by leveraging the functional pattern of known AD-associated genes. Our constructed network outperformed existing networks in predicting AD genes. We then systematically validated the predictions using independent genetic, transcriptomic, proteomic data, neuropathological and clinical data. First, top-ranked genes were enriched in AD-associated pathways. Second, using external gene expression data from the Mount Sinai Brain Bank study, we found that the top-ranked genes were significantly associated with neuropathological and clinical traits, including the Consortium to Establish a Registry for Alzheimer's Disease score, Braak stage score and clinical dementia rating. The analysis of Alzheimer's brain single-cell RNA-seq data revealed cell-type-specific association of predicted genes with early pathology of AD. Third, by interrogating proteomic data in the Religious Orders Study and Memory and Aging Project and Baltimore Longitudinal Study of Aging studies, we observed a significant association of protein expression level with cognitive function and AD clinical severity. The network, method and predictions could become a valuable resource to advance the identification of risk genes for AD.

Emotion recognition has become a research focus in the brain–computer interface and cognitive neuroscience. Electroencephalogram (EEG) is employed for its advantages as accurate, objective, and noninvasive nature. However, many existing research only focus on extracting the time and frequency domain features of the EEG signals while failing to utilize the dynamic temporal changes and the positional relationships between different electrode channels. To fill this gap, we develop the dynamic differential entropy and brain connectivity features based EEG emotion recognition using linear graph convolutional network named DDELGCN. First, the dynamic differential entropy feature which represents the frequency domain feature as well as time domain feature is extracted based on the traditional differential entropy feature. Second, brain connectivity matrices are constructed by calculating the Pearson correlation coefficient, phase-locked value and transfer entropy, and then are used to denote the connectivity features of all electrode combinations. Finally, a linear graph convolutional network is customized and applied to aggregate the features from total electrode combinations and then classifies the emotional states, which consists of five layers, namely, an input layer, two linear graph convolutional layers, a fully connected layer, and a softmax layer. Extensive experiments show that the accuracies in the valence and arousal dimensions reach 90.88% and 91.13%, and the precision reaches 96.66% and 97.02% on the DEAP dataset, respectively. On the SEED dataset, the accuracy and precision reach 91.56% and 97.38%, respectively.

Affective brain—computer interfaces have become an increasingly important topic to achieve emotional intelligence in human—machine collaboration. However, due to the complexity of electroencephalogram (EEG) signals and the individual differences in emotional response, it is still a great challenge to design a reliable and effective model. Considering the influence of personality traits on emotional response, it would be helpful to integrate personality information and EEG signals for emotion recognition. This study proposes a personality-guided attention neural network that can use personality information to learn effective EEG representations for emotion recognition. Specifically, we first use a convolutional neural network to extract rich temporal and regional representations of EEG signals, and a special convolution kernel is designed to learn inter- and intra-regional correlations simultaneously. Second, inspired by the fact that electrodes within distinct brain scalp regions play different roles in emotion recognition, a personality-guided regional-attention mechanism is proposed to further explore the contributions of electrodes within a region and between regions. Finally, attention-based long short-term memory is designed to explore the temporal dynamics of EEG signals. Experiments on the AMIGOS dataset, which is a dataset for multimodal research for affect, personality traits, and mood on individuals and groups, show that the proposed method can significantly improve the performance of subject-independent emotion recognition and outperform state-of-the-art methods.

Person re-identification (Re-ID) via gait features within 3D skeleton sequences is a newly-emerging topic with several advantages. Existing solutions either rely on hand-crafted descriptors or supervised gait representation learning. This paper proposes a self-supervised gait encoding approach that can leverage unlabeled skeleton data to learn gait representations for person Re-ID. Specifically, we first create self-supervision by learning to reconstruct unlabeled skeleton sequences reversely, which involves richer high-level semantics to obtain better gait representations. Other pretext tasks are also explored to further improve self-supervised learning. Second, inspired by the fact that motion's continuity endows adjacent skeletons in one skeleton sequence and temporally consecutive skeleton sequences with higher correlations (referred as locality in 3D skeleton data), we propose a locality-aware attention mechanism and a locality-aware contrastive learning scheme, which aim to preserve locality-awareness on intra-sequence level and inter-sequence level respectively during self-supervised learning. Last, with context vectors learned by our locality-aware attention mechanism and contrastive learning scheme, a novel feature named Constrastive Attention-based Gait Encodings (CAGEs) is designed to represent gait effectively. Empirical evaluations show that our approach significantly outperforms skeleton-based counterparts by 15-40 percent Rank-1 accuracy, and it even achieves superior performance to numerous multi-modal methods with extra RGB or depth information. Our codes are available at https://github.com/Kali-Hac/Locality-Awareness-SGE.

Drug addicts are characterized by difficulty neglecting monetary reward, but its underlying neural mechanisms remain unclear. The current study aimed to investigate the behavioral and electrophysiological signatures of abnormal attentional bias based on different amounts of reward in abstinent heroin addicts (AHAs). We used a modified attentional capture task while recording EEG in 18 AHAs and 18 age-, gander-, and education-matched healthy controls (HCs). We analyzed the attentional distribution of the relative positional changes in space of the target and reward-related stimulus. When targets integrated reward-related colors, participants were more responsive and deployed more attention to targets, especially those with high-value colors. When targets and reward-related distractors were spatially separated, high-value distractors captured the AHA's attention and slowed their responses. Moreover, AHAs had weaker attentional control than HCs, exhibiting an inability to suppress the attentional bias driven by high-value stimuli. Overall, these results demonstrated that AHAs was hypersensitive to task-irrelevant and previous reward-related stimuli, possibly due to damage to brain reward circuits caused by chronic heroin abuse. Our work provides novel behavioral and neurophysiological evidence that are closely associated with the maintenance and relapse of addiction.

Depression is a mental disorder characterized by persistent low mood that affects a person’s thoughts, behavior, feelings, and sense of well-being. According to the World Health Organization (WHO), depression will become the second major life-threatening illness in 2020. Electroencephalogram (EEG) signals, which reflect the working status of human brain, are regarded as the best physiological tool for depression detection. Previous studies used the Empirical Mode Decomposition (EMD) method, which can deal with the highly complex, nonlinear and non-stationary nature of EEG, to extract features from EEG signals. However, for some special data, the neighboring components extracted through EMD could certainly have sections of data carrying the same frequency at different time durations. Thus, the Intrinsic Mode Functions (IMFs) of the data could be linearly dependent and the features coefficients of expansion based on IMFs could not be extracted, which can make the pre-proposed EMD-based feature extraction method impractical. In order to solve this problem, an improved EMD applying Singular Value Decomposition (SVD)-based feature extraction method was proposed in this study, which can extract the features coefficients of expansion based on all IMFs as accurately as possible, ignoring potentially linear dependence of IMFs. Experiments were conducted on four EEG databases for detecting depression. The improved EMD-based feature extraction method can extract feature from all three channels (Fp1, Fpz, and Fp2) on the four EEG databases. The average classification results of the proposed method on the four EEG databases including depressed patients and healthy subjects reached 83.27, 85.19, 81.98 and 88.07 percent, respectively, which were comparable with the pre-proposed EMD-based feature extraction method.

Electroencephalograph (EEG) emotion recognition plays an important role in the brain-computer interface (BCI) field. However, most of recent methods adopted shallow graph neural networks using a single temporal feature, leading to the limited emotion classification performance. Furthermore, the existing methods generally ignore the individual divergence between different subjects, resulting in poor transfer performance. To address these deficiencies, we propose a dual-branch dynamic graph convolution based adaptive transformer feature fusion network with adapter-finetuned transfer learning (DBGC-ATFFNet-AFTL) for EEG emotion recognition. Specifically, a dual-branch graph convolution network (DBGCN) is firstly designed to effectively capture the temporal and spectral characterizations of EEG simultaneously. Second, the adaptive Transformer feature fusion network (ATFFNet) is conducted by integrating the obtained feature maps with the channel-weight unit, leading to significant difference between different channels. Finally, the adapter-finetuned transfer learning method (AFTL) is applied in cross-subject emotion recognition, which proves to be parameter-efficient with few samples of the target subject. The competitive experimental results on three datasets have shown that our proposed method achieves the promising emotion classification performance compared with the state-of-the-art methods. The code of our proposed method will be available at:https://github.com/smy17/DANet.

Electroencephalogram (EEG) has been widely used for the detection of anxiety because of its ability to reflect the functional activities of the brain. However, EEG alone may not provide precision in the detection of anxiety because other emotional disorders usually trigger the same changes in brain function. To discover effective diagnostic indicators and to achieve more precise anxiety detection, we integrate eye movement information into EEG and divide the features into groups according to their respective characteristics. Then, we use group sparse canonical correlation analysis (GSCCA) to investigate group structure information among EEG and eye movement features and obtain an effective fusion representation of EEG and eye movement to achieve more precise detection of anxiety mood. The experimental results from 45 anxious subjects and 47 normal controls from the Healthy Brain Network (HBN) dataset showed that GSCCA could be effectively used to explore the correlation between EEG features within different scalp regions and eye movement features from several aspects. Visual behaviors, including saccades and fixation, are more linearly related to the power spectrum of EEG on the scalp area corresponding to the visual region of the brain. The ultimate fusion representation achieved an optimal classification accuracy of 82.70 percent with the support vector machine (SVM) classifier on the gamma band of EEG.

Welcome to the fifth issue of IEEE Transactions on Computational Social Systems (TCSS) in 2022. After the usual introduction of our 24 regular articles, we would like to discuss the topic of “Psychological Field Versus Physiological Field: From Qualitative Analysis to Quantitative Modelling of the Mental Status.”

Welcome to the second issue of IEEE Transactions on Computational Social Systems (TCSS) in 2022. In this issue, we are going to present 25 regular articles. After the “scanning the issue,” I would like to share some of my opinions and perspectives on the fundamentals of computational psychophysiology: theory and methodology.

Welcome to the fourth issue of IEEE Transactions on Computational Social Systems (TCSS) in 2022. First, we have some exciting news to share. In late June, Clarivate updated the Impact Factor of all journals which are indexed by Web of Science. According to the Journal Citation Reports, the 2021 Journal Impact Factor of IEEE TCSS was 4.727. Many thanks to all for your great effort and support.

Welcome to the last issue of IEEE Transactions On Computational Social Systems (TCSS) of 2022. In this issue, we publish a Special Issue on Advanced Cognitive Computing for Data-Driven Computational Social Systems, which includes 23 articles. Moreover, we also would like to share some of our opinions and perspectives on “Digital Mental Health—Breaking a Lance for Prevention.”

Welcome to the third issue of IEEE Transactions on Computational Social Systems (TCSS) of 2022. According to the latest update of CiteScoreTracker from Elsevier Scopus released on April 6, 2022, the CitesSore of IEEE TCSS has reached a historical high of 8.4. Many thanks to all for your great effort and support.

Welcome to the first issue of the IEEE Transactions on Computational Social Systems (TCSS) of 2022. We would like to take this opportunity to express our sincere thanks to our associate editors, reviewers, authors, and readers for your great support and effort devoted to IEEE TCSS. Happy New Year to you all, and cheers to health, happiness, and high-producing in 2022!

Graph theory analysis using electroencephalogram (EEG) signals is currently an advanced technique for seizure prediction. Recent deep learning approaches, which fail to fully explore both the characterizations in EEGs themselves and correlations among different electrodes simultaneously, generally neglect the spatial or temporal dependencies in an epileptic brain and, thus, produce suboptimal seizure prediction performance consequently. To tackle this issue, in this article, a patient-specific EEG seizure predictor is proposed by using a novel spatio-temporal-spectral hierarchical graph convolutional network with an active preictal interval learning scheme (STS-HGCN-AL). Specifically, since the epileptic activities in different brain regions may be of different frequencies, the proposed STS-HGCN-AL framework first infers a hierarchical graph to concurrently characterize an epileptic cortex under different rhythms, whose temporal dependencies and spatial couplings are extracted by a spectral-temporal convolutional neural network and a variant self-gating mechanism, respectively. Critical intrarhythm spatiotemporal properties are then captured and integrated jointly and further mapped to the final recognition results by using a hierarchical graph convolutional network. Particularly, since the preictal transition may be diverse from seconds to hours prior to a seizure onset among different patients, our STS-HGCN-AL scheme estimates an optimal preictal interval patient dependently via a semisupervised active learning strategy, which further enhances the robustness of the proposed patient-specific EEG seizure predictor. Competitive experimental results validate the efficacy of the proposed method in extracting critical preictal biomarkers, indicating its promising abilities in automatic seizure prediction.

The importance of detecting whether a person wears a face mask while speaking has tremendously increased since the outbreak of SARS-CoV-2 (COVID-19), as wearing a mask can help to reduce the spread of the virus and mitigate the public health crisis. Besides affecting human speech characteristics related to frequency, face masks cause temporal interferences in speech, altering the pace, rhythm, and pronunciation speed. In this regard, this paper presents two effective neural network models to detect surgical masks from audio. The proposed architectures are both based on Convolutional Neural Networks (CNNs), chosen as an optimal approach for the spatial processing of the audio signals. One architecture applies a Long Short-Term Memory (LSTM) network to model the time-dependencies. Through an additional attention mechanism, the LSTM-based architecture enables the extraction of more salient temporal information. The other architecture (named ConvTx) retrieves the relative position of a sequence through the positional encoder of a transformer module. In order to assess to which extent both architectures can complement each other when modelling temporal dynamics, we also explore the combination of LSTM and Transformers in three hybrid models. Finally, we also investigate whether data augmentation techniques, such as, using transitions between audio frames and considering gender-dependent frameworks might impact the performance of the proposed architectures. Our experimental results show that one of the hybrid models achieves the best performance, surpassing existing state-of-the-art results for the task at hand.

Currently, depression has become a common mental disorder, especially among postgraduates. It is reported that postgraduates have a higher risk of depression than the general public, and they are more sensitive to contact with others. Thus, a non-contact and effective method for detecting people at risk of depression becomes an urgent demand. In order to make the recognition of depression more reliable and convenient, we propose a multi-modal gait analysis-based depression detection method that combines skeleton modality and silhouette modality. Firstly, we propose a skeleton feature set to describe depression and train a Long Short-Term Memory (LSTM) model to conduct sequence strategy. Secondly, we generate Gait Energy Image (GEI) as silhouette features from RGB videos, and design two Convolutional Neural Network (CNN) models with a new loss function to extract silhouette features from front and side perspectives. Then, we construct a multi-modal fusion model consisting of fusing silhouettes from the front and side views at the feature level and the classification results of different modalities at the decision level. The proposed multi-modal model achieved accuracy at 85.45% in the dataset consisting of 200 postgraduate students (including 86 depressive ones), 5.17% higher than the best single-mode model. The multi-modal method also shows improved generalization by reducing the gender differences. Furthermore, we design a vivid 3D visualization of the gait skeletons, and our results imply that gait is a potent biometric for depression detection.

It is critical to reduce the operating cost of shared buses for bus companies and improve the user experience of passengers. However, existing studies focus on either bus scheduling or route planning, which cannot accomplish the above mentioned goals concurrently. In this paper, we construct a joint bus scheduling and route planning framework to maximize the number of passengers, minimize the total length of routes and the number of required buses, as well as guarantee good user experience of passengers. First, we establish a system model based on a real-world scenario and formulate a multi-objective combinational optimization problem. Then, based on the extracted traffic topology of urban traffic networks and the generated candidate line set, we propose an offline algorithm to cope with the similar passenger flow distributions, e.g., morning or evening peak of every day. In order to cope with dynamic real-time passenger flows, an online algorithm is designed. Experiments are carried out based on real-word scenarios. The results show that the proposed algorithms can greatly reduce the operating cost of bus companies and guarantee good user experience based on real-world scheduling data in comparison with several existing methods.

A variety of novel mobile applications are developed to attract the interests of potential users in the emerging 5G-enabled vehicular networks. Although computation offloading and task scheduling have been widely investigated, it is rather challenging to decide the optimal offloading ratio and perform adaptive task scheduling in high-dynamic networks. Furthermore, the scheduling policy made by the network operator may be violated, since vehicular users are rational and selfish to maximize their own profits. By considering the incentive compatibility and individual rationality of vehicular users, we present POETS, an efficient partial computation offloading and adaptive task scheduling algorithm to maximize the overall system-wide profit. Specially, a two-sided matching algorithm is first proposed to derive the optimal transmission scheduling discipline. After that, the offloading ratio of vehicular users can be obtained through convex optimization, without any information of other users. Furthermore, a non-cooperative game is constructed to derive the payoff of vehicular users that can reach the equilibrium between users and the network operator. Theoretical analyses and performance evaluations based on real-world traces of taxies demonstrate the effectiveness of our proposed solution.

Intelligent Transportation System (ITS) is critical to cope with traffic events, e.g., traffic jams and accidents, and provide services for personal traveling. However, existing researches have not jointly considered the user data safety, utility and system latency comprehensively, to the best of our knowledge. Since both safe and efficient transmissions are significant for ITS, we construct a blockchain-enabled crowdsensing framework for distributed traffic management. First, we illustrate the system model and formulate a multi-objective optimization problem. Due to its complexity, we decompose it into two subproblems, and propose the corresponding schemes, i.e., a Deep Reinforcement Learning (DRL)-based algorithm and a DIstributed Alternating Direction mEthod of Multipliers (DIADEM) algorithm. Extensive experiments are carried out to evaluate the performance of our solutions, and experimental results demonstrate that the DRL-based algorithm can legitimately select active miners and transactions to make a satisfied trade-off between the blockchain safety and latency, and the DIADEM algorithm can effectively select task computation modes for vehicles in a distributed way to maximize their social welfare.

It is generally considered that controlling a robot precisely becomes tough on the condition of unknown structure information. Applying a data-driven approach to the robot control with the unknown structure implies a novel feasible research direction. Therefore, in this article, as a combination of the structural learning and robot control, an acceleration-level data-driven repetitive motion planning (DDRMP) scheme is proposed with the corresponding recurrent neural network (RNN) constructed. Then, theoretical analyses on the learning and control abilities are provided. Moreover, simulative experiments on employing the acceleration-level DDRMP scheme as well as the corresponding RNN to control a Sawyer robot and a Baxter robot with unknown structure information are performed. Accordingly, simulation results validate the feasibility of the proposed method and comparisons among the existing repetitive motion planning (RMP) schemes indicate the superiority of the proposed method. This work offers sufficient theoretical and simulative solutions for the acceleration-level redundancy problem of redundant robots with unknown structure and joint limits considered.

Sleep inertia is a transitional state from sleep to wakefulness, accompanied by groggy feelings and cognitive impairment. Previous research on sleep inertia mainly used expensive and cumbersome equipment, and the analysis of physiological signals relied on computers. This work introduces a sleep inertia detection system that consists of a wearable low-power electroencephalogram (EEG) acquisition module based on STM32WB55 and ADS1299, and a data processing module based on the Xilinx®Zynq®-7000 XC7Z020. This work recorded the EEG signals of ten subjects in the alert and sleep inertia states to extract the delta power, alpha power, beta power, EEG vigilance, and sample entropy. A linear support vector machine (SVM) was then used to classify the two states based on all subjects’ EEG signals, with an accuracy of 72.5%, and the average accuracy based on a single participant was 8S.9%. Finally, the feature extraction algorithm and SVM parameters were entered into the Zynq®system-on-chip (SoC) development board to realize onboard processing of the algorithm. The system is capable of evaluating the severity of human sleep inertia, which has reference significance for the practical application of sleep inertia detection.

Depression is one of the most common mental disorders affecting people, but its recognition rate is low due to subjectivity and other factors. Mild depression, in particular, has milder and less recognizable symptoms. In this study, we used electroencephalography (EEG) and machine learning methods to identify mild depression. Our approach was to construct a new weighted combinatorial classifier model to distinguish patients with mild depression from normal controls. In this experiment, 10 mildly depressed patients and 10 normal controls watched different emotional facial pictures, recorded their EEG signals and preprocessed them, and then extracted linear and nonlinear features to construct feature vectors. K-nearest neighbor(KNN), support vector machine(SVM), logistic regression(LR), random forest(RF) and back propagation neural network(BPNN) were selected as five individual classifiers, and the differential evolution algorithm(DE) was used to optimize the weights to improve the overall performance of the recognition model. The experimental results showed that the classification accuracy of the proposed method was better than that of the individual classifiers, and the highest 99.09% was achieved when the number of iterations was 50, indicating that the fusion model had a higher recognition accuracy of mild depression than the single modes. At the same time, compared with other combination strategies, this method was also better than other strategies. This research may provide a means to identify mild depression.

Depression is a common mood disorder, which has the characteristics of high prevalence, high recurrence rate, high disability rate and high mortality rate. There are a large number of medical literature on depression, but the number is large and disorderly, which will undoubtedly increase the burden of biomedical researchers and medical workers to obtain knowledge, and is not conducive to the research on the pathogenesis and treatment of depression. Therefore, we construct a knowledge graph of depression based on biomedical literature to assist the study of depression. We use medical abstracts as the main data source and extract knowledge from them by using SemRep, which is a biomedical information extraction system. Secondly, we use another information extraction tool named OpenIE to correct the data extracted by SemRep. Then, by fusing the extracted knowledge with structured data extracted from SemMedDB, we finally get 8,840 triples which include 3,055 entities and 30 relationships. We store them into the graph database Neo4j to visualize the knowledge graph.

Emotion recognition is a key technique of intelligent human-computer interaction (HCI) systems. In the current research on emotion recognition, there are several limitations such as inconsistent brain network scale and high time complexity of modal decomposition. To overcome these shortcomings, we propose a novel emotion recognition method incorporating MST-based brain network and FVMD-GAMPE. Firstly, electroencephalography (EEG) data is decomposed into four frequency bands $(\theta,\alpha,\beta,\gamma)$ by wavelet packet transform (WPT), and mutual information (MI) between channel pairs is calculated to construct the connectivity matrix. Secondly, the brain network based on the minimum spanning tree (MST) is constructed and seven features are extracted. Thirdly, fast variational modal decomposition (FVMD) and WPT are applied to process EEG data to obtain the variational mode functions (VMF) of different frequency bands. Then, the parameters of the multi-scale permutation entropy (MPE) are optimized with the genetic algorithm (GA), and then MPE features are extracted. Finally, the features extracted from MST-based brain network are fused with MPE features, and then fused features are fed to the random forest (RF) classifier to recognize emotional states. Experimental results on DEAP show that the best classification accuracy for valance and arousal are 89.58% and 88.54%, respectively. The result analysis demonstrates MST-based brain network in the negative emotional states has a more divergent topology. This means that brain regions are more active and have a faster exchange of information flow when the brain processes negative emotions. On the other hand, brain network of women is similar to a star-shaped structure, which indicates women’s brain activation is higher than man. This study provides theoretical support for research on negative bias.

Emotion recognition has become a research focus in the field of human-computer interaction (HCI). As an excellent physiological signal, electroencephalographic (EEG) is considered to be a favorable tool for emotion recognition. Most traditional methods focus on extracting features in time domain and frequency domain but the adjacent information and asymmetric information from adjacent and asymmetric channels are often ignored. Although several graph neural network (GNN) models are utilized to learn EEG features, most of the emotion recognition studies of GNN ignore the information existing between adjacent electrodes. In this paper, we propose an EEG emotion recognition method based on hierarchy graph convolution network (HGCN) named ERHGCN. Firstly, six different features including power spectral density (PSD), differential entropy (DE), differential asymmetry (DASM), rational asymmetry (RASM), asymmetry (ASM) and differential caudality (DCAU) from five frequency bands are extracted. Secondly, to improve graph convolution network (GCN) shortcoming of only extracting time and frequency features, HGCN is applied to extract deeper spatial feature by treating the longitudinal and transverse adjacent electrode pairs in different ways. Finally, six extracted features are fed into the HGCN model, then all features are integrated by two full connection layers. We conducted extensive experiments on DEAP dataset and experimental results show that the proposed method can obtain 90.56% and 88.79% recognition accuracies for valence and arousal classification tasks.

Skeleton-based person re-identification (Re-ID) is an emerging open topic providing great value for safety-critical applications. Existing methods typically extract hand-crafted features or model skeleton dynamics from the trajectory of body joints, while they rarely explore valuable relation information contained in body structure or motion. To fully explore body relations, we construct graphs to model human skeletons from different levels, and for the first time propose a Multi-level Graph encoding approach with Structural-Collaborative Relation learning (MG-SCR) to encode discriminative graph features for person Re-ID. Specifically, considering that structurally-connected body components are highly correlated in a skeleton, we first propose a multi-head structural relation layer to learn different relations of neighbor body-component nodes in graphs, which helps aggregate key correlative features for effective node representations. Second, inspired by the fact that body-component collaboration in walking usually carries recognizable patterns, we propose a cross-level collaborative relation layer to infer collaboration between different level components, so as to capture more discriminative skeleton graph features. Finally, to enhance graph dynamics encoding, we propose a novel self-supervised sparse sequential prediction task for model pre-training, which facilitates encoding high-level graph semantics for person Re-ID. MG-SCR outperforms state-of-the-art skeleton-based methods, and it achieves superior performance to many multi-modal methods that utilize extra RGB or depth features. Our codes are available at https://github.com/Kali-Hac/MG-SCR.

Person re-identification via 3D skeletons is an emerging topic with great potential in security-critical applications. Existing methods typically learn body and motion features from the body-joint trajectory, whereas they lack a systematic way to model body structure and underlying relations of body components beyond the scale of body joints. In this paper, we for the first time propose a Self-supervised Multi-scale Skeleton Graph Encoding (SM-SGE) framework that comprehensively models human body, component relations, and skeleton dynamics from unlabeled skeleton graphs of various scales to learn an effective skeleton representation for person Re-ID. Specifically, we first devise multi-scale skeleton graphs with coarse-to-fine human body partitions, which enables us to model body structure and skeleton dynamics at multiple levels. Second, to mine inherent correlations between body components in skeletal motion, we propose a multi-scale graph relation network to learn structural relations between adjacent body-component nodes and collaborative relations among nodes of different scales, so as to capture more discriminative skeleton graph features. Last, we propose a novel multi-scale skeleton reconstruction mechanism to enable our framework to encode skeleton dynamics and high-level semantics from unlabeled skeleton graphs, which encourages learning a discriminative skeleton representation for person Re-ID. Extensive experiments show that SM-SGE outperforms most state-of-the-art skeleton-based methods. We further demonstrate its effectiveness on 3D skeleton data estimated from large-scale RGB videos. Our codes are open at https://github.com/Kali-Hac/SM-SGE.

The rapid development of the COVID-19 pandemic has threatened the lives of people around the world. Many people were caught in anxiety and panic, which also prevents people from fully concentrating on their normal lives. However, the current common neurofeedback therapies used to solve the problem of lack of attention cannot fully deal with the differences in each individual. In addition, direct contact between the patient and the doctor also increases the risk of virus transmission during treatment. This article combines neurofeedback and IoT to establish an adaptive attention adjustment method. IoT connects patients and doctors remotely, reducing the direct contact between them. In order to adapt to individual differences, the feedback indicators of each individual are individually calibrated. In addition, the proportional, integral, and derivative controller was used to adjust the difficulty of the feedback task to adapt to each individual’s self-regulation ability and provide the individual with a higher level of regulation. We also designed adaptive attention adjustment experiments for different individuals. The results show that through adaptive feedback training, the individual’s feedback indicator has dropped by 77.90%, and the individual can adjust his attention state to the individual’s optimal baseline threshold, and the oscillation error gradually reduces to the expected threshold range. This method can cope with the differences between different individuals and provide each individual with the same level of feedback regulation. In the future, this study may provide a general adjuvant treatment for other mental illnesses.

The Internet of Medical Things (IoMT) is an extension and specialization of that original Internet of Things (IoT) concept, and applies to the interconnectedness of devices, software applications, and data which are specific to the medical industry. IoMT can add smart technologies to medical devices to monitor the progression of a disease away from the doctor’s office and learn things that could impact future care guidelines and patients. It can also provide a better way to care for our elderly by tracking vitals and heart performance, glucose and other body systems, and activity and sleeping levels. During the outbreak of pandemic (e.g., COVID-19), IoMT can even be used to detect main symptoms ubiquitously using intelligent sensors and trace the origin of the outbreak based on aggregated IoT data (e.g., geographic mobile data and purchase history). Although most of the contemporary IoMT systems can measure risks, make decisions, and take actions automatically, the lack of emotion-aware abilities will be an obstacle to more harmonious human–machine interaction and more efficient medical process. Besides, mental disorders, such as depression, schizophrenia, and anxiety, have become a more noticeable cause of suffering. The integration of emotion-aware abilities into IoMT can also contribute to monitor emotional dysregulation continuously in subjects with mental disorders or undergoing serious pandemic such as COVID-19, and give these patients personalized therapy recommendations. Research on affective computing has defined a framework to recognize, interpret, and process human affects, but more research is needed to investigate its application to biomedical applications, especially “in the wild” and over extended periods of time, and how to integrate emotion-aware abilities into IoMT organically is still an open question. This special issue aims to create a platform for researchers, developers, and practitioners from both academia and industry to disseminate the state-of-the-art results and to advance the Emotion-Aware ubiquitous computing in IoMT.

Action recognition is essential for many human-centered applications in the Internet of Things (IoT). Especially, in the Internet of Medical Things (IoMT), action recognition shows great importance in surgical assistance, patient monitoring, etc. Recently, 3-D skeleton sequence-based action recognition draws broad attention. It is a challenging task that needs effective modeling on intraframe skeleton representations and interframe temporal dynamics. Standard long short-term memory (LSTM)-based models are widely used for sequence modeling due to its long-term memory, yet they are unable to fully model the relationship between different body joints or persons to extract crucial co-occurrence features from different levels. To handle this shortcoming, we propose an attention-based multilevel co-occurrence graph convolutional LSTM (AMCGC-LSTM). By integrating graph convolutional networks (GCNs) into LSTM, the proposed model is capable of leveraging body structural information from skeletons and strengthening the multilevel co-occurrence (MC) feature learning. Specifically, we first design the spatial attention module for feature enhancement of key joints from skeleton inputs. Second, we design MC memory units coupled with GCN to automatically model the spatial relationship between joints, and simultaneously capture the co-occurrence features from different joints, persons, and frames. Finally, we construct aggregated features of MCs (AFMCs) from MC memory units to better represent the intraframe action context encoding, and leverage a concurrent LSTM (Co-LSTM) to further model their temporal dynamics for action recognition. Our model significantly outperforms mainstream methods on NTU RGB+D 60/120 data set, mutual action subset of NTU RGB+D 60/120 data set, and Northewestern-UCLA data set.

The Internet of Medical Things (IoMT) aims to exploit the Internet-of-Things (IoT) techniques to provide better medical treatment scheme for patients with smart, automatic, timely, and emotion-aware clinical services. One of the IoMT instances is applying IoT techniques to sleep-aware smartphones or wearable devices' applications to provide better sleep healthcare services. As we all know, sleep is vital to our daily health. What is more, studies have shown a strong relationship between sleep difficulties and various diseases such as COVID-19. Therefore, leveraging IoT techniques to develop a longer lifetime sleep healthcare IoMT system, with a tradeoff between data transferring/processing speed and battery energy efficiency, to provide longer time services for bad sleep condition persons, especially the COVID-19 patients or survivors, is a meaningful research topic. In this study, we propose an IoT-enabled sleep data fusion networks (SDFN) module with a star topology Bluetooth network to fuse data of sleep-aware applications. A machine learning model is built to detect sleep events through an audio signal. We design two data reprocessing mechanisms running on our IoT devices to alleviate the data jam problem and save the IoT devices' battery energy. The experiments manifest that the presented module and mechanisms can save the energy of the system and alleviate the data jam problem of the device.

The prompt evolution of Internet of Medical Things (IoMT) promotes pervasive in-home health monitoring networks. However, excessive requirements of patients result in insufficient spectrum resources and communication overload. Mobile Edge Computing (MEC) enabled 5G health monitoring is conceived as a favorable paradigm to tackle such an obstacle. In this paper, we construct a cost-efficient in-home health monitoring system for IoMT by dividing it into two sub-networks, i.e., intra-Wireless Body Area Networks (WBANs) and beyond-WBANs. Highlighting the characteristics of IoMT, the cost of patients depends on medical criticality, Age of Information (AoI) and energy consumption. For intra-WBANs, a cooperative game is formulated to allocate the wireless channel resources. While for beyond-WBANs, considering the individual rationality and potential selfishness, a decentralized non-cooperative game is proposed to minimize the system-wide cost in IoMT. We prove that the proposed algorithm can reach a Nash equilibrium. In addition, the upper bound of the algorithm time complexity and the number of patients benefiting from MEC is theoretically derived. Performance evaluations demonstrate the effectiveness of our proposed algorithm with respect to the system-wide cost and the number of patients benefiting from MEC.

Autism spectrum disorder (ASD) is accompanied with impaired social-emotional functioning, such as emotional regulation and recognition, communication, and related behavior. Study of the alternations of the brain networks in ASD may not only help us in understanding this disorder but also inform us the mechanisms of affective computing in the brain. Although morphological features have been used in the diagnosis of a variety of neurological and psychiatric disorders, these features did not show significant discriminative value in identifying patients with ASD, possibly due to the omission of the information related to the changes in structural similarities among cortical regions. In this study, structural images from 66 high-functioning adults with ASD and 66 matched typically-developing controls (TDC) were used to test the hypothesis of cortico-cortical relationships are abnormal in ASD. Seven morphological features of each of the 360 brain regions were extracted and elastic network was used to quantify the similarities between each target region and all other regions. The similarities were then used to construct multi-feature-based networks (MFN), which were then submitted to a support vector machine classifier to classify the individuals of the two groups. Results showed that the classifier with features of MFN significantly improved the accuracy of discriminating patients with ASD from TDCs (78.63 percent) compared to using morphological features only (<; 65 percent). The combination of MFN features with morphological features and other high-level MFN properties did not further enhance the classification performance. Our findings demonstrate that the variations in cortico-cortical similarities are important in the etiology of ASD and can be used as biomarkers in the diagnostic process.

Welcome to the first issue of IEEE Transactions on Computational Social Systems (TCSS) of 2021. I am excited to serve as the new Editor-in-Chief (EiC) of our journal. First, I would like to thank my predecessor, Prof. Fei-Yue Wang from the Chinese Academy of Science, for his dedication and commitment over the past three years. Under his outstanding leadership, along with the contribution and effects from our associate editors and supporting staff, IEEE TCSS is becoming a thriving journal of the IEEE Systems, Man, and Cybernetics Society (SMC). I also would like to express my appreciation to the IEEE Publication Committee members and my colleagues at the IEEE SMC for putting their trust in me.

Welcome to the second issue of IEEE Transactions on Computational Social Systems (TCSS) this year. First, I am grateful to report that, as of February 7, 2021, theCitescoreof TCSS has leapfrogged back to 5.8, a new high, which indicates the high quality and relevance of IEEE TCSS in the field of social computing and computational social systems research. Many thanks to all of you for your great effort and support.

Welcome to the last issue of IEEE Transactions on Computational Social Systems (IEEE TCSS) in 2021. For IEEE TCSS, 2021 is an exciting year. TCSS has been added to the ISI Web of Science Sources Citation Index Expanded (SCIE) database in 2021, and all articles published since 2018 have been indexed by SCIE. This is an important milestone in the development of TCSS. We would like to take this opportunity to thank and congratulate everyone for their great efforts and supports. We are looking forward to working together to further improve the publication quality and speed up the review process of TCSS in the upcoming 2022.

Welcome to the fifth issue of Ieee Transactions on Computational Social Systems (TCSS) of 2021. As usual, we would like to share some great news first. Since April 2021, IEEE TCSS has been added to the Science Citation Index Expanded (SCIE) database in Clarivate Web of Science. We are excited to report that all TCSS articles published since 2018 have been backtracked and indexed by SCIE.

Welcome to the fourth issue of IEEE Transactions on Computational Social Systems (TCSS) this year. I am excited to share some great news.

Federated ecology can provide an effective solution forthe serious isolated data island issues caused by data privacy protection and information security requirements in the era of artificial intelligence (AI). As the data foundation of federated ecology, federated data include the data of all the nodes in the federation, as wellas their storage, computation, and communication resources. For privacy-preserving, federated data are divided into private data and non-private data, and through the federated control of these data, data federalization can be realized. In data-driven AI technologies, federated data play an important role, and it can help realize effective data retrieval, pre-processing, processing, mining, and visualization for AI-based applications. It can also provide effective solutionsfor the dilemmas faced by AI technologies, such as training AI models without sufficient data, increasing the generality of AI models for different application scenarios and establishing a unified processing workflow for data security and privacy control in AI-based applications.

These days, physiological signals have been studied more broadly for emotion recognition to realize emotional intelligence in human-computer interaction. However, due to the complexity of emotions and individual differences in physiological responses, how to design reliable and effective models has become an important issue. In this article, we propose a regularized deep fusion framework for emotion recognition based on multimodal physiological signals. After extracting the effective features from different types of physiological signals, we construct ensemble dense embeddings of multimodal features using kernel matrices, and then utilize a deep network architecture to learn task-specific representations for each kind of physiological signal from these ensemble dense embeddings. Finally, a global fusion layer with a regularization term, which can efficiently explore the correlation and diversity among all of the representations in a synchronous optimization process, is designed to fuse generated representations. Experiments on two benchmark datasets show that this framework can improve the performance of subject-independent emotion recognition compared to single-modal classifiers or other fusion methods. Data visualization also demonstrates that the final fusion representation exhibits higher class-separability power for emotion recognition.

Driver fatigue has become a leading cause of accidents and death in the transportation industry. Electroencephalography (EEG)-based fatigue detection can be a good way to reduce accidents and improve safety and efficiencies throughout the transportation system. In this article, we focus on investigating whether the spatial–temporal changes in the relations between EEG channels are specific to different driving states. EEG signals were first partitioned into several segments, and the covariance matrices obtained from each segment were input into a recurrent neural network to extract high-level temporal features. Meanwhile, the covariance matrices of whole signals were leveraged to extract spatial characteristics that were fused with temporal features to obtain comprehensive spatial–temporal information. In experiments on an open benchmark dataset, our method achieved an excellent classification accuracy of 89.28% and showed superior performance compared to several other state-of-the-art methods. These results indicate that our method can enable higher performance in driver fatigue detection.

Depression is a mental disorder with emotional and cognitive dysfunction. The main clinical characteristic of depression is significant and persistent low mood. As reported, depression is a leading cause of disability worldwide. Moreover, the rate of recognition and treatment for depression is low. Therefore, the detection and treatment of depression are urgent. Multichannel electroencephalogram (EEG) signals, which reflect the working status of the human brain, can be used to develop an objective and promising tool for augmenting the clinical effects in the diagnosis and detection of depression. However, when a large number of EEG channels are acquired, the information redundancy and computational complexity of the EEG signals increase; thus, effective channel selection algorithms are required not only for machine learning feasibility, but also for practicality in clinical depression detection. Consequently, we propose an optimal channel selection method for EEG-based depression detection via kernel-target alignment (KTA) to effectively resolve the abovementioned issues. In this method, we consider a modified version KTA that can measure the similarity between the kernel matrix for channel selection and the target matrix as an objective function and optimize the objective function by a proposed optimal channel selection strategy. Experimental results on two EEG datasets show that channel selection can effectively increase the classification performance and that even if we rely only on a small subset of channels, the results are still acceptable. The selected channels are in line with the expected latent cortical activity patterns in depression detection. Moreover, the experimental results demonstrate that our method outperforms the state-of-the-art channel selection approaches.

Recently, Internet of Vehicles (IoV) has become one of the most active research fields in both academic and industry, which exploits resources of vehicles and Road Side Units (RSUs) to execute various vehicular applications. Due to the increasing number of vehicles and the asymmetrical distribution of traffic flows, it is essential for the network operator to design intelligent offloading strategies to improve network performance and provide high-quality services for users. However, the lack of global information and the time-variety of IoVs make it challenging to perform effective offloading and caching decisions under long-term energy constraints of RSUs. Since Artificial Intelligence (AI) and machine learning can greatly enhance the intelligence and the performance of IoVs, we push AI inspired computing, caching and communication resources to the proximity of smart vehicles, which jointly enable RSU peer offloading, vehicle-to-RSU offloading and content caching in the IoV framework. A Mix Integer Non-Linear Programming (MINLP) problem is formulated to minimize total network delay, consisting of communication delay, computation delay, network congestion delay and content downloading delay of all users. Then, we develop an online multi-decision making scheme (named OMEN) by leveraging Lyapunov optimization method to solve the formulated problem, and prove that OMEN achieves near-optimal performance. Leveraging strong cognition of AI, we put forward an imitation learning enabled branch-and-bound solution in edge intelligent IoVs to speed up the problem solving process with few training samples. Experimental results based on real-world traffic data demonstrate that our proposed method outperforms other methods from various aspects.

Recent developments of edge computing and content caching in wireless networks enable the Intelligent Transportation System (ITS) to provide high-quality services for vehicles. However, a variety of vehicular applications and time-varying network status make it challenging for ITS to allocate resources efficiently. Artificial intelligence algorithms, owning the cognitive capability for diverse and time-varying features of Internet of Connected Vehicles (IoCVs), enable an intent-based networking for ITS to tackle the above-mentioned challenges. In this paper, we develop an intent-based traffic control system by investigating Deep Reinforcement Learning (DRL) for 5G-envisioned IoCVs, which can dynamically orchestrate edge computing and content caching to improve the profits of Mobile Network Operator (MNO). By jointly analyzing MNO's revenue and users' quality of experience, we define a profit function to calculate the MNO's profits. After that, we formulate a joint optimization problem to maximize MNO's profits, and develop an intelligent traffic control scheme by investigating DRL, which can improve system profits of the MNO and allocate network resources effectively. Experimental results based on real traffic data demonstrate our designed system is efficient and well-performed.

Effective features can improve the performance of a model and help us understand the characteristics and underlying structure of complex data. Previously proposed feature selection methods usually cannot retain more discriminative information. To address this shortcoming, we propose a novel supervised orthogonal least square regression model with feature weighting for feature selection. The optimization problem of the objective function can be solved by employing generalized power iteration and augmented Lagrangian multiplier methods. Experimental results show that the proposed method can more effectively reduce feature dimensionality and obtain better classification results than traditional feature selection methods. The convergence of our iterative method is also proved. Consequently, the effectiveness and superiority of the proposed method are verified both theoretically and experimentally.

The explosive growth of mobile devices promotes the prosperity of novel mobile applications, which can be realized by service offloading with the assistance of edge computing servers. However, due to limited computation and storage capabilities of a single server, long service latency hinders the continuous development of service offloading in mobile networks. By supporting multi-server cooperation, Pervasive Edge Computing (PEC) is promising to enable service migration in highly dynamic mobile networks. With the objective of maximizing the system utility, we formulate the optimization problem by jointly considering the constraints of server storage capability and service execution latency. To enable dynamic service placement, we first utilize Lyapunov optimization method to decompose the long-term optimization problem into a series of instant optimization problems. Then, a sample average approximation-based stochastic algorithm is proposed to approximate the future expected system utility. Afterwards, a distributed Markov approximation algorithm is utilized to determine the service placement configurations. Through theoretical analysis, the time complexity of our proposed algorithm is linear to the number of users, and the backlog queue of PEC servers is stable. Performance evaluations are conducted based on both synthetic and real trace-driven scenarios, with numerical results demonstrating the effectiveness of our proposed algorithm from various aspects.

Electroencephalogram(EEG) and eye movement have been extensively applied in the detection of anxiety disorders because they can reflect the brain functions and people's attentional bias. Although our previous work can make good use of the group structure information of EEG and eye movement signals, it mainly models the linear correlation and ignores the nonlinear correlation between two modalities. Therefore, we proposed kernel group sparse canonical correlation analysis (K-GSCCA) to study the nonlinear complex relationship and group structure information among EEG and eye movement features. Firstly, EEG signals were divided into 13 groups according to different brain regions, and eye movement signals were divided into 4 groups according to different visual behaviors. Then, we used the Gaussian kernel function to transform data into kernel space, effectively generated nonlinear cooperative fusion representation. The experimental outcomes demonstrated that K-GSCCA can be effective to solve the nonlinear correlation of group structure information between EEG and eye movement features. Using the support vector machine(SVM) classifier, we finally achieved the best classification accuracy of 87.47% in the fusion of the gamma band of EEG and eye movement features.

Judging mental fatigue can be guided by acquiring and analyzing various physiological data. However, the existing equipment focuses on single physiological signals, such as electroencephalogram (EEG) and electrocardiogram (ECG), which ignores the preponderance of multimodal physiological signals in fatigue states. Alternatively, some equipment is difficult to operate and thus unsuitable for portable applications. To this end, this paper details the design of a miniaturized multi-physiological signal acquisition system. The system not only acquires EEG and ECG based on combining wavelet transform with Kalman filter, but also uses precise temperature sensors to synchronously acquire proximal skin temperature signals which are not easily interfered with by environmental noise or other physiological signals of human body. Through a fatigue assessment experiment on ten healthy subjects, it was verified that our equipment reliably detects mental fatigue states by monitoring and analyzing EEG, ECG, and proximal skin temperature data. In actual applications, this system appears to the traits of easy operation and good stability. It provides better hardware support for the pervasive applications and concrete implementation of mental health in multi-scene, and can popularize use.

This Electroencephalography (EEG)-based research is to explore the effective biomarkers for depression recognition. Resting-state EEG data were collected from 24 major depressive patients (MDD) and 29 normal controls using 128-electrode geodesic sensor net. To better identify depression, we extracted multi-type of EEG features including linear features (L), nonlinear features (NL), functional connectivity features phase lagging index (PLI) and network measures (NM) to comprehensively characterize the EEG signals in patients with MDD. And machine learning algorithms and statistical analysis were used to evaluate the EEG features. Combined multi-types features (All: L+ NL + PLI + NM) outperformed single-type features for classifying depression. Analyzing the optimal features set we found that compared to other type features, PLI occupied the largest proportion of which functional connections in intra-hemisphere were much more than that of in inter-hemisphere. In addition, when using PLI features and All features, high frequency bands (alpha, beta) could achieve obviously higher classification accuracy than low frequency bands (delta, theta). Parietal-occipital lobe in the high frequency bands had great effect in depression identification. In conclusion, combined multi-types EEG features along with a robust classifier can better distinguish depressive patients from normal controls. And intra-hemispheric functional connections might be an effective biomarker to detect depression. Hence, this paper may provide objective and potential electrophysiological characteristics in depression recognition.

Although more and more researchers pay attention to the emotion classification, traditional emotion classification methods can not embrace changes in the global and local areas of the human brain after being stimulated. We propose an emotion classification method based on SVM combining brain functional connectivity. Firstly, the nonlinear phase-locked value (PLV) is used to calculate the multiband brain functional connectivity network, which is then converted into a binary brain network, and seven features of binary brain network are calculated. Secondly, support vector machines (SVM) are used to classify positive and negative emotions at the valence dimension and arousal dimension in the multiband. Experimental results on DEAP show that the best emotion classification accuracy of the proposed method is 86.67% in the arousal dimension, and 84.44% in the valence dimension. The results demonstrate that the classification accuracy of the arousal dimension is better than the valence dimension and the Beta2 frequency band is more suitable for emotion classification. Finally, several findings on brain functional connectivity network is discussed. The left and right areas of brain functional connectivity network are unbalanced in the low frequency band, and the feature values of clustering coefficient, average shortest path length, global efficiency, local efficiency, node degree are positively correlated with the arousal degree in the arousal dimension. Humans emotions are suppressed in the low frequency band, and the brain functional connectivity network after emotional stimulation is strengthened in the high frequency band. Our findings on emotion classification are valuable and consistent with the study of neural mechanisms.

Emotion plays an indispensable role in the process of human cognition and decision-making, but it is often neglected in human-computer interaction (HCI). Although many researchers have applied some decomposition algorithms to extract features from EEG, there are several shortcomings including the modal aliasing, a high computational cost and so on. In this paper, we introduce variational mode decomposition (VMD) and wavelet packet decomposition (WPD) algorithms and propose a multidimensional feature extraction method based on VMD and WPD for emotion recognition. Firstly, we apply VMD to decompose the EEG signal into a specific number of variational mode functions (VMF). Secondly, WPD is executed to generate an emotional frequency band. Then, we continue to extract the wavelet packet entropy (WPE), modified multi-scale sample entropy (MMSE), fractal dimension (FD) and first difference (1ST) of each emotional VMF to construct a new feature form. Finally, the random forest (RF) is utilized to classify the extracted emotional states. The experiment results demonstrate that the proposed method is more competitive and universal for emotion recognition, which provides a novel idea for the application of multidimensional feature extraction.

Conventional hyper-network coefficients ignore the weighted hyper-edge information which could be vital in researching the specificity of brain disease. Functional hyper-networks for 64 healthy controls (HC) and 56 patients with major depressive disorder (MDD) were constructed using the least absolute shrinkage and selection operator (Lasso). Not only the classical topology metrics but also a novel hyper-edge weight (HEW) attribute were extracted as features to promote the functional-based auto-diagnosis accuracy of MDD. We compared the categorization performance of each hyper-network coefficient. A multi-feature ensemble model was applied to fuse different kinds of features. We obtained 82.15 % accuracy with the classical hyper-network clustering coefficient (HCC) and 84.08 % accuracy with the HEW attribute on the MDD dataset. The performance was further improved to 89.24% by combining all the properties of the hyper-networks. The multi-feature ensemble model combining different hyper-network coefficients provides new insights into the automatic diagnosis with diverse information of MDD.

As the challenge of depression problems increases today, it is important to effective and timely detect for patients' treatments and depression prevention. Current methods of automated depression diagnosis depend almost entirely on audio, video, and Electroencephalogram (EEG) etc. In this paper, we propose a novel method to detect depression using gait data that is collected by Kinect camera. Its key components include a camera rectification method and a rigid-body representation of the human body. The camera rectification method achieves the purpose of improving data processing accuracy by changing the camera angle. The rigid-body representation can not only improve the robustness of detecting depression patients with noisy input, but also can reduce the classification time. We evaluate our method on the depression gait dataset in postgraduate students. The proposed method has an outstanding performance in classic machine learning algorithms, and the best accuracy can achieve 88.89%. Our solution provides a new method for automatic depression detection (ADD) that has exciting implications in clinical theory and practice, and has the advantages of high accuracy, inexpensive, low time cost, and no-contact.

Human action recognition plays an important role in e-health applications, such as surgical skill analysis, patient monitoring, and automatic nursing systems. Recently, skeleton-based action recognition gains massive attention. It is an essential yet challenging task that requires effectively modeling the intra-frame skeleton representation and inter-frame temporal dynamics. Traditional Long Short-Term Memory (LSTM) based methods mainly capture long-term action context information from global level, yet they cannot fully model the relationship between different joints or persons to mine crucial co-occurrence features from different levels. To overcome this drawback, we propose a general end-to-end Multi-level Co-occurrence Graph Convolutional LSTM (MCGC-LSTM). By incorporating graph convolutional networks (GCN) into LSTM, our model can not only better exploit body structural information from skeletons but also enhance the multi-level co-occurrence feature learning. Specifically, we first devise multi-level co-occurrence (MC) memory units coupled with GCN to automatically model the spatial relationship between joints, and simultaneously capture the co-occurrence features from different joints, persons, and frames. Then we construct aggregated features of multi-level co-occurrences (AFMC) from MC memory units to better represent the intra-frame action context encoding, and leverage a concurrent LSTM (Co-LSTM) to further model their temporal dynamics for action recognition. Experiments show that our proposed model significantly outperforms mainstream methods on NTU RGB+D 120 dataset and Northwestern-UCLA dataset.

Generalized anxiety disorder (GAD), as one of the most common chronic anxiety disorders, faces difficulties in clinical diagnosis. With the rapid development and wide application of smartphones in recent years, smartphones have a vivid application prospect in the field of mental disease monitoring and diagnosis. Based on WeChat applet platform on smartphones, an APP that integrates scale testing and inertial sensor data collection is developed to study the detection of subjects with GAD in task state. A behavior patterns extraction method is proposed using sliding windows to split behavior data, and processing data segments for clustering. Distribution information are extracted from the subjects' behavior patterns and are combined with the descriptive statistical features of the sample to identify GAD. The results show that this method has an accuracy of 66.44% for female subjects and 71.43% for male subjects in GAD recognition.

Gait-based person re-identification (Re-ID) is valuable for safety-critical applications, and using only 3D skeleton data to extract discriminative gait features for person Re-ID is an emerging open topic. Existing methods either adopt hand-crafted features or learn gait features by traditional supervised learning paradigms. Unlike previous methods, we for the first time propose a generic gait encoding approach that can utilize unlabeled skeleton data to learn gait representations in a self-supervised manner. Specifically, we first propose to introduce self-supervision by learning to reconstruct input skeleton sequences in reverse order, which facilitates learning richer high-level semantics and better gait representations. Second, inspired by the fact that motion's continuity endows temporally adjacent skeletons with higher correlations (“locality”), we propose a locality-aware attention mechanism that encourages learning larger attention weights for temporally adjacent skeletons when reconstructing current skeleton, so as to learn locality when encoding gait. Finally, we propose Attention-based Gait Encodings (AGEs), which are built using context vectors learned by locality-aware attention, as final gait representations. AGEs are directly utilized to realize effective person Re-ID. Our approach typically improves existing skeleton-based methods by 10-20% Rank-1 accuracy, and it achieves comparable or even superior performance to multi-modal methods with extra RGB or depth information.

Automatic real-time emotion recognition based on multi-channel EEG signals is a significant and challenging task in neurology and psychiatry. In recent years, deep learning has been used in EEG emotion recognition. However, many existing deep learning based methods still require complex pre-processing or additional feature extraction, which make it difficult to achieve real-time emotion recognition. In this paper, an end-to-end model named Temporal Convolutional Broad Learning System (TCBLS) was designed for multi-channel EEG based emotion recognition. The TCBLS takes one-dimensional EEG signals as input, then extracts emotion-related features of EEG automatically. In this model, the Temporal Convolutional Network (TCN) is designed to extract EEG temporal features and deep abstract features simultaneously, then Broad Learning System (BLS) is used to map the features to a more discriminative space and further enhance the features. We evaluated our method on DEAP database, performing 10-fold cross-validation on each subject to obtain the classification accuracy. Experimental results indicate that the performance of TCBLS is better than other comparison methods, and the mean accuracy of TCBLS is 99.5755% and 99.5781% on valence and arousal classification task respectively. The results demonstrate the effectiveness and robustness of TCBLS in EEG emotion recognition.

Depression, threatening the well-being of millions, has become one of the major diseases in the past decade. However, the current method of diagnosing depression is questionnaire-based interviews, which is labor-intensive and highly dependent on doctors' experience. Thus, objective and cost-efficient methods are needed. In this paper, we present a case-based reasoning model for identifying depression. Electroencephalography data were collected using a portable three-electrode EEG device, and then processed to remove artifacts and extract features. We applied multiple classifiers. The best performing k-Nearest Neighbor (KNN) was selected as the evaluation function to select the effective features which were then used to create the case base. Based on the weight set of standard deviations, the similarity was calculated using normalized Euclidean distance to get the optimal recognition rate of depression. The accuracy of optimal similarity identification of patients with depression was 91.25 percent, which was improved compared to the accuracy using KNN classifier (81.44 percent) or previously reported classifiers. Thus, we provide a novel pervasive and effective method for automatic detection of depression.

Recently, microexpression recognition has attracted a lot of researchers' attention due to its challenges and valuable applications. However, it is noticed that currently most of the existing proposed methods are often evaluated and tested on the single database and, hence, this brings us a question whether these methods are still effective if the training and testing samples belong to different domains, for example, different microexpression databases. In this case, a large feature distribution difference may exist between training (source) and testing (target) samples and, hence, microexpression recognition tasks would become more difficult. To solve this challenging problem, that is, cross-domain microexpression recognition, in this paper, we propose an effective method consisting of an auxiliary set selection model (ASSM) and a transductive transfer regression model (TTRM). In our method, an ASSM is designed to automatically select an optimal set of samples from the target domain to serve as the auxiliary set, which is used for subsequent TTRM training. As for TTRM, it aims at bridging the feature distribution gap between the source and target domains by learning a joint regression model with the source domain samples and the auxiliary set selected from the target domain. We evaluate the proposed TTRM plus ASSM by extensive cross-domain microexpression recognition experiments on SMIC and CASME II databases. Compared with the recent state-of-the-art domain adaptation methods, our proposed method has a more satisfactory performance in dealing with the cross-domain microexpression recognition tasks.

The emerging 5G-enabled vehicular networks can satisfy various requirements of vehicles by traffic offloading. However, limited cellular spectrum and energy supplies restrict the development of 5G-enabled applications in vehicular networks. In this article, we construct an intelligent offloading framework for 5G-enabled vehicular networks, by jointly utilizing licensed cellular spectrum and unlicensed channels. A cost minimization problem is formulated by considering the latency constraint of users and is further decomposed into two subproblems due to its complexity. For the first subproblem, a two-sided matching algorithm is proposed to schedule the unlicensed spectrum. Then, a deep-reinforcement-learning-based method is investigated for the second one, where the system state is simplified to realize distributed traffic offloading. Real-world traces of taxies are leveraged to illustrate the effectiveness of our solution.

The public's purchase incentive increases dramatically during the holiday season and subsequently returns to normal levels. This seasonality is common in various scenarios and highlights the following questions: how does the public's purchase incentive fluctuate over the course of a year? Which factors are conducive to this seasonal behavior and how can they be modeled? In this paper, we propose a model that explicitly integrates temporal point process theory with the construction of a networked community, to describe the dynamics of collective action propagation with seasonal fluctuation. Furthermore, a database is constructed of sales records for 21 video game consoles and 13 237 video games in France, Germany, Japan, the U.K., the USA, and worldwide from 1989 to 2018. Experimental results suggest that peak desire always appears in the holiday season about one week before Christmas and is about four times higher than consumption desire in a normal period in all areas.

Depression is a mental disorder characterized by emotional and cognitive dysfunction, which appears a state of low mood and aversion to activity. Depression can affect a person's thoughts, behavior, feelings, and sense of well-being. Depression is projected to be the second major life-threatening illness in 2020 by World Health Organization (WHO). Thus, it is urgent to detect and treat depression. Electroencephalogram (EEG) signals, which objectively reflect the working status of the human brain, are considered as promising physiological tools for depression detection. Negatively biased processing of affective stimuli in depression has been proven. In order to detect depression more effectively, we proposed an affective auditory stimuli induced depression detection method from EEG signals. In this method, we applied negative, positive and neutral affective auditory stimuli with several frequency selected from the International Affective Digitized Sounds (IADS-2) to induce negative affective bias in patients with depression. We synchronously collected EEG signals with three electrodes located on the prefrontal lobe (Fpl, Fpz, and Fp2), then extracted efficacious features by Empirical Mode Decomposition (EMD) based feature extraction method to detect depression effectively. The results of the proposed method showed that high-frequency affective auditory stimuli were more effective in depression detection and the frequency of affective auditory stimuli was a crucial property, which can influence the effectiveness of affective auditory stimuli in depression detection.

Depression, influencing millions of people, has become a major disease in the past decade. However, the assessment methods of diagnosing depression almost exclusively rely on patient-reported or clinical judgments of symptom severity, which are associated with subjective biases and intensive labor. Some bio-signals such as EEG and eye movements are used for automatic detection but their accuracies are not accurate enough for the real application, further improvements are needed. This research proposes a content based ensemble method (CBEM) to promote the depression detection accuracy, generating data subsets by the content of the experiment, then using the majority vote of subsets to determine the subjects' label. The validation of the method is testified by two different experiments which included free viewing eye tracking and task-state EEG and these two experiments have 36, 40 subjects respectively. In these two experiments CBEM gains accuracies of 82.5% and 92.73% respectively. The results show that CBEM outperform traditional classification methods. Our findings provide an effective solution for promoting the accuracy of depression identification, and give an objective and quantitative evaluation of depression, which in the future could be used for the auxiliary diagnosis of depression.

Mobile edge computing (MEC) has risen as a promising paradigm to provide high quality of experience via relocating the cloud server in close proximity to smart mobile devices (SMDs). In MEC networks, the MEC server with computation capability and storage resource can jointly execute the latency-sensitive offloading tasks and cache the contents requested by SMDs. In order to minimize the total latency consumption of the computation tasks, we jointly consider computation offloading, content caching, and resource allocation as an integrated model, which is formulated as a mixed integer nonlinear programming (MINLP) problem. We design an asymmetric search tree and improve the branch and bound method to obtain a set of accurate decisions and resource allocation strategies. Furthermore, we introduce the auxiliary variables to reformulate the proposed model and apply the modified generalized benders decomposition method to solve the MINLP problem in polynomial computation complexity time. Simulation results demonstrate the superiority of the proposed schemes.

It has been found that there exists an essential similarity between solving equations and controlling dynamic systems: Both errors are expected to decrease to zero (or an acceptably tiny value) as soon as possible. By exploiting such a similarity, researchers have presented and investigated continuous-time recurrent neural network models for solving time-varying problems. To be compatible with digital computers, it is desirable to develop discrete-time neural algorithms from the control perspective for performance improvement. In this paper, a discrete-time zeroing neural algorithm is proposed for the solving system of linear equations with the aid of control techniques. To lay a basis for theoretical analyses, the proposed zeroing neural algorithm with nonlinearity is converted into a second-order linear system plus a residual term, and then, analyzed using the control theory. Theoretical results and numerical experiments are provided, which illustrate that the proposed neural algorithm possesses an improved performance compared to the existing solutions.

Currently, vehicles have the abilities to communicate with each other autonomously. For Internet of Vehicles (IoV), it is urgent to reduce the latency and improve the throughput for data transmission among vehicles. This article proposes a deep learning based transmission strategy by exploring trirelationships among vehicles. Specifically, we consider both the social and physical attributes of vehicles at the edge of IoV, i.e., edge of vehicles. The social features of vehicles are extracted to establish the network model by constructing triangle motif structures to obtain primary neighbors with close relationships. Additionally, the connection probabilities of nodes based on the characteristics of vehicles and devices can be estimated, by which a content sharing partner discovery algorithm is proposed based on convolutional neural network. Finally, the experiment results demonstrate the efficiency of our method with respect to various aspects, such as message delivery ratio, average latency, and percentage of connected devices.

Recurrent neural networks (RNNs) have found a great variety of application areas. As a special type of RNNs, zeroing neural network (ZNN), or termed Zhang neural network, has been reported to have powerful abilities to address various nonstationary problems. To overcome drawbacks and improve the performance of existing ZNN models, several modified ZNN models are proposed in this paper, which allow nonconvex activation functions and possess accelerated finite-time convergence property. Theoretical analyses suggest that the developed ZNN models are equipped with the global convergence property and the convergence-accelerated models are verified by the estimated upper bounds of convergence time. Finally, comparative and illustrative simulation results, including a verification on a mobile manipulator, are presented to illustrate the effectiveness and superiority of proposed ZNN models to existing models for solving nonstationary Sylvester equations.

Currently, depression has become a common mental disorder and one of the main causes of disability worldwide. Due to the difference in depressive symptoms evoked by individual differences, how to design comprehensive and effective depression detection methods has become an urgent demand. This study explored from physiological and behavioral perspectives simultaneously and fused pervasive electroencephalography (EEG) and vocal signals to make the detection of depression more objective, effective and convenient. After extraction of several effective features for these two types of signals, we trained six representational classifiers on each modality, then denoted diversity and correlation of decisions from different classifiers using co-decision tensor and combined these decisions into the ultimate classification result with multi-agent strategy. Experimental results on 170 (81 depressed patients and 89 normal controls) subjects showed that the proposed multi-modal depression detection strategy is superior to the single-modal classifiers or other typical late fusion strategies in accuracy, f1-score and sensitivity. This work indicates that late fusion of pervasive physiological and behavioral signals is promising for depression detection and the multi-agent strategy can take advantage of diversity and correlation of different classifiers effectively to gain a better final decision.

Because of the enormous potential to guarantee road safety and improve driving experience, social Internet of Vehicle (SIoV) is becoming a hot research topic in both academic and industrial circles. As the ever-increasing variety, quantity, and intelligence of on-board equipment, along with the evergrowing demand for service quality of automobiles, the way to provide users with a range of security-related and user-oriented vehicular applications has become significant. This paper concentrates on the design of a service access system in SIoVs, which focuses on a reliability assurance strategy and quality optimization method. First, in lieu of the instability of vehicular devices, a dynamic access service evaluation scheme is investigated, which explores the potential relevance of vehicles by constructing their social relationships. Next, this work studies a trajectory-based interaction time prediction algorithm to cope with an unstable network topology and high rate of disconnection in SIoVs. At last, a cooperative quality-aware system model is proposed for service access in SIoVs. Simulation results demonstrate the effectiveness of the proposed scheme.

As an emerging communication platform, opportunistic Cloud of Things (CoT) is promising for clients to exchange messages through opportunistic contacts in cloud computing-enabled Internet of Things. Recently, numerous socially aware schemes have been put forward, leveraging users’ social attributes and contact history to predict future contacts with the purpose of improving message forwarding efficiency and network throughput. However, individual privacy is generally overlooked in the prediction process and transmission stage of opportunistic CoT. In this paper, we construct a privacy-preserving message forwarding framework for opportunistic CoT to guarantee individual privacy and improve transmission efficiency. We first set up a two-layer architecture of a cloud server to improve communication efficiency for terminal clients. By integrating a security-based mobility prediction algorithm with a routing decision process, our scheme can effectively protect individual privacy. We integrate an attribute-based cryptographic algorithm with a message delivery process to enable our scheme to resist attacks, such as Sybil attack, drop for profit, and data tampered attack. Compared with some existing solutions, our scheme improves network security significantly at the cost of slightly increased communication overhead.

Mobile edge computing (MEC) brings computation capacity to the edge of mobile networks in close proximity to smart mobile devices (SMDs) and contributes to energy saving compared with local computing, but resulting in increased network load and transmission latency. To investigate the tradeoff between energy consumption and latency, we present an energy-aware offloading scheme, which jointly optimizes communication and computation resource allocation under the limited energy and sensitive latency. In this paper, single and multicell MEC network scenarios are considered at the same time. The residual energy of smart devices' battery is introduced into the definition of the weighting factor of energy consumption and latency. In terms of the mixed integer nonlinear problem for computation offloading and resource allocation, we propose an iterative search algorithm combining interior penalty function with D.C. (the difference of two convex functions/sets) programming to find the optimal solution. Numerical results show that the proposed algorithm can obtain lower total cost (i.e., the weighted sum of energy consumption and execution latency) comparing with the baseline algorithms, and the energy-aware weighting factor is of great significance to maintain the lifetime of SMDs.

In sharing economy, people offer idle social resources to others in a sharing manner. Through community-based online platforms, the people offering services can earn commission while others can enjoy a better life via renting social resources. Consequently, the value-in-use of services is expectedly strengthened within the unit time, although the total amount of social resources remains constant. Influenced by sharing economy, some famous companies have developed intelligent systems to analyze the most appropriate coincidence between citizens' idle supply and renting demand from numerous data sets. However, the big data analysis of the optimal service-demand matching usually runs on the traditional multiprocessors equipped in intelligent systems, so-called “system-on-chip.” In this paper, we design a novel computer architecture - the accelerator based on optical network-on-chip (ONoC) - to further speed up the matching between citizens' offer and demand in sharing economy. Our ONoC-based accelerator is able to quickly calculate the optimal service-demand matching by processing computation tasks on parallel cores, i.e., task-core mapping. In addition, to improve the accelerator reliability, the assorted task-core mapping algorithm is also designed. The extensive simulation results based on real trace file demonstrate the effectiveness of our system and algorithm. Note to Practitioners - Sharing economy is of great importance for realizing green consumption and sustainable development in our human society. Sharing economy enterprise calls for intelligent system design for service-demand matching in the current big data era. In this paper, we design the accelerator based on ONoC to further speed up the matching between citizens' offer and demand in sharing economy. By processing computation tasks on parallel cores using our algorithm, the task-core mapping can be performed with high speed and reliability. The simulation results - based on the trace file of Amazon Mechanical Turk - can well guide the practitioners to design a more clever and reliable product by quickly calculating the optimal service-demand matching.

High dimensional biomedical datasets contain thousands of features which can be used in molecular diagnosis of disease, however, such datasets contain many irrelevant or weak correlation features which influence the predictive accuracy of diagnosis. Without a feature selection algorithm, it is difficult for the existing classification techniques to accurately identify patterns in the features. The purpose of feature selection is to not only identify a feature subset from an original set of features [without reducing the predictive accuracy of classification algorithm] but also reduce the computation overhead in data mining. In this paper, we present our improved shuffled frog leaping algorithm which introduces a chaos memory weight factor, an absolute balance group strategy, and an adaptive transfer factor. Our proposed approach explores the space of possible subsets to obtain the set of features that maximizes the predictive accuracy and minimizes irrelevant features in high-dimensional biomedical data. To evaluate the effectiveness of our proposed method, we have employed the K-nearest neighbor method with a comparative analysis in which we compare our proposed approach with genetic algorithms, particle swarm optimization, and the shuffled frog leaping algorithm. Experimental results show that our improved algorithm achieves improvements in the identification of relevant subsets and in classification accuracy.

The research detailed in this paper focuses on the processing of Electroencephalography (EEG) data to identify attention during the learning process. The identification of affect using our procedures is integrated into a simulated distance learning system that provides feedback to the user with respect to attention and concentration. The authors propose a classification procedure that combines correlation-based feature selection (CFS) and a k-nearest-neighbor (KNN) data mining algorithm. To evaluate the CFS+KNN algorithm, it was test against CFS+C4.5 algorithm and other classification algorithms. The classification performance was measured 10 times with different 3-fold cross validation data. The data was derived from 10 subjects while they were attempting to learn material in a simulated distance learning environment. A self-assessment model of self-report was used with a single valence to evaluate attention on 3 levels (high, neutral, low). It was found that CFS+KNN had a much better performance, giving the highest correct classification rate (CCR) of % for the valence dimension divided into three classes.

Several anatomical magnetic resonance imaging (MRI) markers for Alzheimer's disease (AD) have been identified. Cortical gray matter volume, cortical thickness, and subcortical volume have been used successfully to assist the diagnosis of Alzheimer's disease including its early warning and developing stages, e.g., mild cognitive impairment (MCI) including MCI converted to AD (MCIc) and MCI not converted to AD (MCInc). Currently, these anatomical MRI measures have mainly been used separately. Thus, the full potential of anatomical MRI scans for AD diagnosis might not yet have been used optimally. Meanwhile, most studies currently only focused on morphological features of regions of interest (ROIs) or interregional features without considering the combination of them. To further improve the diagnosis of AD, we propose a novel approach of extracting ROI features and interregional features based on multiple measures from MRI images to distinguish AD, MCI (including MCIc and MCInc), and health control (HC). First, we construct six individual networks based on six different anatomical measures (i.e., CGMV, CT, CSA, CC, CFI, and SV) and Automated Anatomical Labeling (AAL) atlas for each subject. Then, for each individual network, we extract all node (ROI) features and edge (interregional) features, and denoted as node feature set and edge feature set, respectively. Therefore, we can obtain six node feature sets and six edge feature sets from six different anatomical measures. Next, each feature within a feature set is ranked by -score in descending order, and the top ranked features of each feature set are applied to MKBoost algorithm to obtain the best classification accuracy. After obtaining the best classification accuracy, we can get the optimal feature subset and the corresponding classifier for each node or edge feature set. Afterwards, to investigate the classification performance with only node features, we proposed a weighted multiple kernel learning (wMKL) framework to combine these six optimal node feature subsets, and obtain a combined classifier to perform AD classification. Similarly, we can obtain the classification performance with only edge features. Finally, we combine both six optimal node feature subsets and six optimal edge feature subsets to further improve the classification performance. Experimental results show that the proposed method outperforms some state-of-the-art methods in AD classification, and demonstrate that different measures contain complementary information.

This paper is to design substitution boxes (S-Boxes) using innovative I-Ching operators (ICOs) that have evolved from ancient Chinese I-Ching philosophy. These three operators-intrication, turnover, and mutual- inherited from I-Ching are specifically designed to generate S-Boxes in cryptography. In order to analyze these three operators, identity, compositionality, and periodicity measures are developed. All three operators are only applied to change the output positions of Boolean functions. Therefore, the bijection property of S-Box is satisfied automatically. It means that our approach can avoid singular values, which is very important to generate S-Boxes. Based on the periodicity property of the ICOs, a new network is constructed, thus to be applied in the algorithm for designing S-Boxes. To examine the efficiency of our proposed approach, some commonly used criteria are adopted, such as nonlinearity, strict avalanche criterion, differential approximation probability, and linear approximation probability. The comparison results show that S-Boxes designed by applying ICOs have a higher security and better performance compared with other schemes. Furthermore, the proposed approach can also be used to other practice problems in a similar way.

In this paper, the existing recurrent neural network (RNN) models for solving zero-finding (e.g., matrix inversion) with time-varying parameters are revisited from the perspective of control and unified into a control-theoretical framework. Then, limitations on the activated functions of existing RNN models are pointed out and remedied with the aid of control-theoretical techniques. In addition, gradient-based RNNs, as the classical method for zero-finding, have been remolded to solve dynamic problems in manners free of errors and matrix inversions. Finally, computer simulations are conducted and analyzed to illustrate the efficacy and superiority of the modified RNN models designed from the perspective of control. The main contribution of this paper lies in the removal of the convex restriction and the elimination of the matrix inversion in existing RNN models for the dynamic matrix inversion. This work provides a systematic approach on exploiting control techniques to design RNN models for robustly and accurately solving algebraic equations.

Changes in the motor pattern have been shown to be useful advanced indicators of cognitive disorders, such as Parkinson's disease (PD) and cerebral small vessel disease (SVD). It would be highly advantageous to tap into data containing people's motor patterns from motion sensing devices to analyze subtle changes in cognitive abilities, thereby providing personalized interventions before the actual onset of such conditions. However, this goal is very challenging due to two main technical problems: 1) the size of data labeled by doctors is small, and 2) the available data tends to be highly imbalanced (the vast majority tend to be from normal subjects with only a small fraction from subjects with cognitive disorder). In order to effectively deal with these challenges to infer cognitive wellness from motor patterns with high accuracy, we propose the MOtor-Cognitive Analytics (MOCA) framework. The proposed MOCA first uses the random oversampling iterative random forest based feature selection method to reduce the feature space dimensionality and avoid overfitting, and then adds a bias in the optimization problem of weighted extreme learning machine to achieve good generalization ability in handling imbalanced small-sampling dataset. Experimental results on two real-world datasets including SVD and stroke patients show that MOCA can effectively reduce the rate of misdiagnosis and significantly outperform state-of-the-art methods in inferring people's cognitive capabilities. This work opens up opportunities for population-level pre-screening using motion sensing devices and can inform current discussions on reforming the health-care infrastructure.

Depression is a mental disorder of high prevalence, leading to a negative effect on individuals, their families, society and the economy. In recent years, the problem of automatic detection of depression from the speech signal has gained more interest. In this paper, a new multiple classifier system for depression recognition was developed and tested. The novel aspect of this methodology is the combination of different speech types and emotions. First of all, using a sample of 74 subjects (37 depressed patients and 37 healthy controls), we examined the discriminative power of different speech types (interview, picture description, and reading) and speech emotions (positive, neutral, and negative). Some voice features (e.g. short time energy, intensity, loudness, zero-crossing rate (ZCR), F0, jitter, shimmer, formants, mel frequency cepstral coefficients (MFCC), linear prediction coefficient (LPC), line spectrum pair (LSP), and perceptual linear predictive coefficients (PLP)) were tested. Then, a new multiple classifier method was proposed to detect depression. It was observed that the overall recognition rate using interview speech was higher than employing picture description speech and reading speech. Furthermore, neutral speech showed better performance than positive and negative speech. Among these features, short time energy, ZCR, LPC, MFCC and LSP were the robust features that gave high accuracy in different types of speech. Finally, this new approach showed a high accuracy of 78.02%, giving high encouragement for detecting depression in speech.

As a promising non-invasive technique, functional near-infrared spectroscopy(fNIRS) can easily detect the hemodynamic responses of cortical brain activities. This paper investigated the multiclass classification of motor imagery(MI)based on fNIRS. 10 healthy individuals were recruited to move an object using their imagination. A multi-channel continuous-wave fNIRS equipment was applied to obtain the signals from the prefrontal cortex(PFC). The combined Ensemble Empirical Mode Decomposition (EEMD) and Independent Component Analysis(ICA) method was used to solve the signal-noise frequency spectrum aliasing issues caused by Mayer wave(0.1Hz), then the signal means(SM) features were extracted as an input of Support Vector Machine(SVM) classifier. The average accuracies of 4 directions, up-down and left-right were 40.55%, 73.05%, 70.7% respectively using Hbo2(8-21s). This study demonstrated that Brodmann area 4 was activated, which is consistent with previous conclusions. Furthermore, we found that the orbitofrontal cortex is also involved in MI and O2sat can also serve as a classified index.

Emotion recognition using EEG signals has become a hot research topic in the last few years. This paper aims at providing a novel method for emotion recognition using less channels of frontal EEG signals. By employing the asymmetry theory of frontal brain, a new method fusing spatial and frequency features was presented, which only adopted two channels of frontal EEG signals at Fp1 and Fp2. In order to estimate the efficiency of the method, a GBDT classifier was evaluated and selected, and the method was implemented on the DEAP database. The maximum and mean classification accuracy were achieved as 76.34% and 75.18% respectively, which exhibited the best result comparing with other related studies. This method is extremely suitable for wearable EEG monitoring applications in human daily life.

As the ϵ4 allele of apolipoprotein E (APOE4) is proved a high risk factor of Alzheimer's disease (AD), numerous studies have used modalities of neuroimaging data to investigate the alterations of brain caused by APOE4. A recent study has shown that APOE4-related pathological changes of cortical networks during rest exist in APOE4 carriers. However, the interrelationship among the resting intrinsic networks (ICNs) has not been adequately explored. In the present study, seven ICNs were detected in both of APOE4 carriers and APOE4 non-carriers with spatial independent component analysis (ICA). Then the effective connectivity patterns of the two groups were acquired using multivariate Granger causality analysis (mGCA) the results of which reflect the information flow among ICNs. Compared with APOE4 non-carriers, significant difference in activity was found within default mode network in APOE4 carriers. Moreover, the significantly reduced intensity of causal interaction from auditory network to cerebellum network was found in APOE4 carriers. And we found the significantly causal relationship from cerebellum network to default mode network only existed in the connectivity pattern of APOE4 carriers but not in that of APOE4 non-carriers. In addition, the subcortical network and cognitive control network in APOE4 carriers were found less influenced by other ICNs. The findings in our study suggest that APOE4 indeed modulates the activities and interactions of ICNs, which might be the genetic cause of the dysfunctional process in APOE4 carriers. Our findings may shed new light on the ways how APOE4 affects the functions of nervous systems and provide a perspective to understand genetic basis of pathogenesis of AD.

Recently, Electroencephalogram (EEG) has become increasingly important in the role of psychiatric diagnosis and emotion recognition. However, many irrelevant features make it difficult to identify patterns accurately. Obtaining valid features from electroencephalogram can improve the classification and generalization performance. In this paper, an improved normalized mutual information feature selection algorithm which is based on Grassberger entropy estimator (G-NMIFS) is proposed for EEG data. We employ the k-Nearest Neighbor (kNN), Support Vector Machine (SVM), and Naïve Bayes methods to compare the proposed approach with normalized mutual information feature selection using Naïve estimator and Miller-adjust method. Experimental results on two EEG data sets show that the proposed method can select relevant subsets and improve classification performance effectively.

People spend up to one-third of lives asleep, and healthy sleep habits can make a big difference in their quality of life. But in modern society, many people have unhealthy sleep diaries and suffer from various sleep disorders, which may result in irregular mood fluctuations or even mental health problems such as anxiety and depression. We propose the Emotion-Aware Smart Tips (EAST), a novel approach that could help to inform users about their irregular emotional states with smart tips to improve their sleep qualities. EAST aims at helping users keep healthy sleep schedules and emotional states by providing smart tips through a novel model that combines multivariate regression, random forest, and neural network to quantify the relations between sleep patterns and emotional states. Prototype implementation and initial experiments of EAST in mobile phones have demonstrated its desired functionality and practicality for real-world deployment.

Retrieving objects from large image collection is challenging due to the so-called background-interference, i.e., matching between query object and reference images is usually confused by cluttered background, especially when objects are small. In this paper, we propose an object retrieval technique addressing this problem by partitioning the images. Specifically, several object proposals are partitioned from the images by jointly optimizing their objectness and coverage. The proposal set with maximum objectness score and minimum redundancy is obtained. Therefore,the interference of cluttered background is greatly reduced. Next, the objects are retrieved based on the partitioned proposals, separately and independently to the background. Our method is featured by the fine partitioning, which not only removes interferences from background, but also significantly reduces the number of objects to index. In this way, the effectiveness and efficiency are both achieved, which better suits big data retrieval. Subsequently, feature coding on partitioned objects generates much meaningful representation, and object level connectivity also introduces novel clues into the reranking. Extensive experiments on three popular object retrieval benchmark datasets (Oxford Buildings, Paris, Holiday) show the effectiveness of our method in retrieving small objects out of big data.

In many research and application areas, such as information retrieval and machine learning, we often encounter dealing with a probability distribution that is mixed by one distribution that is relevant to our task in hand and the other that is irrelevant and that we want to get rid of. Thus, it is an essential problem to separate the irrelevant distribution from the mixture distribution. This article is focused on the application in Information Retrieval, where relevance feedback is a widely used technique to build a refined query model based on a set of feedback documents. However, in practice, the relevance feedback set, even provided by users explicitly or implicitly, is often a mixture of relevant and irrelevant documents. Consequently, the resultant query model (typically a term distribution) is often a mixture rather than a true relevance term distribution, leading to a negative impact on the retrieval performance. To tackle this problem, we recently proposed a Distribution Separation Method (DSM), which aims to approximate the true relevance distribution by separating a seed irrelevance distribution from the mixture one. While it achieved a promising performance in an empirical evaluation with simulated explicit irrelevance feedback data, it has not been deployed in the scenario where one should automatically obtain the irrelevance feedback data. In this article, we propose a substantial extension of the basic DSM from two perspectives: developing a further regularization framework and deploying DSM in the automatic irrelevance feedback scenario. Specifically, in order to avoid the output distribution of DSM drifting away from the true relevance distribution when the quality of seed irrelevant distribution (as the input to DSM) is not guaranteed, we propose a DSM regularization framework to constrain the estimation for the relevance distribution. This regularization framework includes three algorithms, each corresponding to a regularization strategy incorporated in the objective function of DSM. In addition, we exploit DSM in automatic (i.e., pseudo) irrelevance feedback, by automatically detecting the seed irrelevant documents via three different document reranking methods. We have carried out extensive experiments based on various TREC datasets, in order to systematically evaluate the proposed methods. The experimental results demonstrate the effectiveness of our proposed approaches in comparison with various strong baselines.

Depression is a common mental disorder, and in recent years, there has been an increasing trend of mild to moderate depression among college students. Additionally, effective detection of mild depression at an earlier stage remains an urgent problem that must be solved. In this study, electroencephalography (EEG) activities were recorded from 37 participants during processing of facial expression stimuli. With both high-gamma and low-gamma bands, the coherence in the right hemisphere of normal controls was greater than that of mildly depressive subjects, especially for electrodes P8, TP8, C4, FC4, and F8. In the low gamma band, the clustering coefficients of healthy controls in the prefrontal lobe (AF4, AFz, AF3, FC5, F4, and F6) and the parietal lobe (PO3, PO4, and P2) were significantly higher than those of mildly depressive subjects. The ratio of the characteristic path length between the functional network of the mildly depressed group and the small-world network was greater than 1. For the normal group, the ratio was near 1. This research contributes to the study of the cognitive process of mild depression. In our study, the results show closer cooperation in the brain areas of right hemisphere in normal controls during the cognitive process compared with the mildly depressed group, while the activity of the prefrontal and parietal regions in mild depression was significantly lower than that of normal controls. At the same time, in terms of the characteristic path length, the functional network of the mildly depressed group deviates from the small-world network.

According to the World Health Organization, it is predicted that in 2020, depression will become the second largest illness threatening the health of mankind. In order to alleviate the worldwide damage caused by depression, a portable and accurate diagnosing technique is the most essential. This research uses three-electrode pervasive EEG collector to collect EEG data from Fp1, Fp2, and Fpz as locations of scalp electrodes, since these locations are closely related to emotions, and uncovered by hair. Special designed experiment has been conducted and totally 178 subjects' EEG data have been collected. Then the research uses KNN (k-Nearest Neighbor), SVM (Support Vector Machine), ANN (Artificial Neuro Network) and DBN (Deep Belief Network) to analyze the data. The results show DBN performed better than traditional methods using shallow algorithms. Moreover, the results suggested the absolute power of beta wave is a valid characteristic, which could be used for detection of depression. The accuracy reached 78.24% using the combination of DBN and the absolute power of beta wave. This research proves the feasibility of smaller-size pervasive system for depression diagnosis.

This paper investigates the potential of physiological signals as reliable channels for multi-subject emotion recognition. A three-stage decision framework is proposed for recognizing four emotions of multiple subjects. The decision framework consists of three stages: (1) in the initial stage, identifying a subject group that a test subject can be mapped to; (2) in the second stage, identifying an emotion pool that an instance of the test subject can be assigned to; and (3) in the final stage, generating the predicted emotion from the given emotion pool for the test instance. In comparison with a series of alternative methods, the high accuracy of 70.04% achieved by our proposed method clearly demonstrates the potential of the three-stage decision method in multi-subject emotion recognition.

Electroencephalogram (EEG) is a noninvasive method to record electrical activity of brain and it has been used extensively in research of brain function due to its high time resolution. However raw EEG is a mixture of signals, which contains noises such as Ocular Artifact (OA) that is irrelevant to the cognitive function of brain. To remove OAs from EEG, many methods have been proposed, such as Independent Components Analysis (ICA), Discrete Wavelet Transform (DWT), Adaptive Noise Cancellation (ANC) and Wavelet Packet Transform (WPT). In this paper, we present a novel hybrid de-noising method which uses Discrete Wavelet Transform (DWT) and Kalman Filtering to remove OAs in EEG. Firstly, we used this method on simulated data. The Mean Squared Error (MSE) of DWT-Kalman method was 0.0017, significantly lower compared to results using WPT-ICA and DWT-ANC, which were 0.0468 and 0.0052, respectively. Meanwhile, the Mean Absolute Error (MAE) using DWT-Kalman achieved an average of 0.0052, which also performed better than WPT-ICA and DWT-ANC, which were 0.0218 and 0.0115, respectively. Then we applied the proposed approach to the raw data collected by our prototype three-channel EEG collector and 64-channel Braincap from BRAIN PRODUCTS. On both data, our method achieved satisfying results. This method does not rely on any particular electrode or the number of electrodes in certain system, so it is recommended for ubiquitous applications.

Summary form only given. Computational psychophysiology is a new direction that broadens the field of psychophysiology by allowing for the identification and integration of multimodal signals to test specific models of mental states and psychological processes. Additionally, such approaches allows for the extraction of multiple signals from large-scale multidimensional data, with a greater ability to differentiate signals embedded in background noise. Further, these approaches allows for a better understanding of the complex psychophysiological processes underlying brain disorders such as autism spectrum disorder, depression, and anxiety. Given the widely acknowledged limitations of psychiatric nosology and the limited treatment options available, new computational models may provide the basis for a multidimensional diagnostic system and potentially new treatment approaches.

Automatic emotion recognition based on multi-channel neurophysiological signals, as a challenging pattern recognition task, is becoming an important computer-aided method for emotional disorder diagnoses in neurology and psychiatry. Traditional approaches require designing and extracting a range of features from single or multiple channel signals based on extensive domain knowledge. This may be an obstacle for non-domain experts. Moreover, traditional feature fusion method can not fully utilize correlation information between different channels. In this paper, we propose a preprocessing method that encapsulates the multi-channel neurophysiological signals into grid-like frames through wavelet and scalogram transform. We further design a hybrid deep learning model that combines the `Convolutional Neural Network (CNN)' and `Recurrent Neural Network (RNN)', for extracting task-related features, mining inter-channel correlation and incorporating contextual information from those frames. Experiments are carried out, in a trial-level emotion recognition task, on the DEAP benchmarking dataset. Our results demonstrate the effectiveness of the proposed methods, with respect to the emotional dimensions of Valence and Arousal.

Various problems caused by stress seriously affect individuals' physical and mental well-being and have been receiving an increasing attention in modern lives. Since traditional stress assessment methods are lack of objectivity, affective sensing technologies have been studied for years. As detecting stress in speech has the advantages of non-invasive, portable, fast, and less expensive, many explorations were conducted to build stress assessment models. To find out a proper acoustic feature subset for a specific reading pattern, we performed the experiments with 30 subjects by three reading patterns: vowel, figure and sentence. We utilized feature selection and classification techniques to automatically select acoustic features and evaluate performances. Results showed that there are interactions between reading patterns and stress levels on speech features. Although Stress levels can be distinguished in any pattern of them (vowel, figure, sentence), sentence is a better choice with the best classification accuracy 88.15%. Furthermore, Line Spectral Pairs (LSP) features are indispensable for vowel, Mel-Frequency Cepstral Coefficient (MFCC) features are more effective for figure and the combination of prosodic, LSP and MFCC features is more suitable for sentence.

Recent studies suggested that cognitive impairments and memory difficulties in cancer survivors were associated with topology changes of brain network, particularly in terms of the functional and structural abnormalities. However, little is known about the modular reconfiguration of metabolic brain network among this population. In this study, we recruited 78 patients with pre-treatment cancer and 80 age- and gender-matched normal controls (NCs), and constructed the metabolic brain networks derived from resting-state 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) to assess the alters of modularity pattern in cancer. The measurements of the participation index (PI) and mutual information (MI) were calculated for the cancer and NC groups. Compared with NC group, one module composed by the hippocampus, the amygdala and frontal and temporal regions was absented in cancer group. Moreover, cancer patients showed abnormal topology pattern in their metabolic networks (i.e., increased local efficiency and reduced global efficiency). Although node-wise PI shared positive correlated with normalized metabolism uptake in both groups, the more energy consumption were observed in metabolism network of cancer group that might be indicative of reduced capability of information processing. In addition, the between-group MIs were gradually increased over a range of thresholds. Our results suggested that modular pattern of the metabolic brain network seemed to re-shape its organization in cancer, which might uncover the neurobiological mechanisms underlying cancer-related cognitive dysfunction.

The rapid development of neuroimaging technology and brain network analysis methodologies have promoted the research of Alzheimer's disease (AD). Recently, studies on brain networks reported that AD patients showed abnormal connectivity alterations and disrupted coordinated organizations compared with normal controls (NC). However, much less knowledge is about the abnormalities of metabolic network at individual level, which might be the potential marker in promoting current AD diagnosis. In the present study, we constructed the individual metabolic network based on 18F-Fluro-Deoxyglucose Positron Emission Tomography (18F-FDG-PET) data by using cubes consisted with certain numbers of voxels. Network properties, connectivity strength and metabolic cost of cubes of 111 NCs and 111 AD patients were calculated to evaluate the performance and feasibility of the proposed network via machine learning approaches. Results showed that the features we extracted were well-performed in classification, with accuracy of 95.64% and area of 0.9915 under receiver operating characteristic curve, indicating the individual metabolic network and local metabolic information are potential powerful in AD diagnosis.

It has been reported that chronic heroin intake induces changes in central nervous system of human brain; however, few studies investigate the carry-over adverse effects on brain after heroin withdrawal. In this work we examined the alpha rhythms of resting-state Electroencephalogram (EEG) signals to measure the neuroelectrical differences between the heroin addicts after heroin withdrawal and normal control. Eyes-closed resting EEG signals from 20 heroin addicts with the abstinence length ranging from 4-16 months and 20 normal controls were recorded using 64 electrodes. Comparing the nonlinear characteristics of EEG signals, such as the correlation dimension, Kolmogorov entropy and Lempel-Ziv complexity, we found that the EEG signals from heroin addicts were significantly more irregular than those from normal controls. Furthermore, the topography of the each nonlinear feature was examined, and the abnormal changes were widely spread over the brain. These findings suggest that nonlinear methods may contribute to gain new insights into brain dysfunction in heroin addicts even after heroin abstinence.

Depression is a common mental disorder with growing prevalence, however current diagnoses of depression face the problem of patient denial, clinical experience and subjective biases from self-report. Our study aims to develop an objective approach to depression detection that supports the process of diagnosis and assists the monitoring of risk factors. By classifying eye movement features during free viewing tasks, an accuracy of 80.1% was achieved using Random Forest to discriminate depressed and nondepressed subjects. Results indicate that eye movement features hold the potential to form a complimentary method of detection, having a relatively low computation overhead. Furthermore, given the proliferation of cheap internet eye movement detection technologies, the method offers the possibility of cost effective remote sensing of the patient mental state.

Depression is a common mental disorder and one of the main causes of disability worldwide. Lacking objective depressive disorder assessment methods is the key reason that many depressive patients can't be treated properly. Developments in affective sensing technology with a focus on acoustic features will potentially bring a change due to depressed patient's slow, hesitating, monotonous voice as remarkable characteristics. Our motivation is to find out a speech feature set to detect, evaluate and even predict depression. For these goals, we investigate a large sample of 300 subjects (100 depressed patients, 100 healthy controls and 100 high-risk people) through comparative analysis and follow-up study. For examining the correlation between depression and speech, we extract features as many as possible according to previous research to create a large voice feature set. Then we employ some feature selection methods to eliminate irrelevant, redundant and noisy features to form a compact subset. To measure effectiveness of this new subset, we test it on our dataset with 300 subjects using several common classifiers and 10-fold cross-validation. Since we are collecting data currently, we have no result to report yet.

Objective: This paper aims to use multimodal physiological signals to automatically recognize human emotions, and a novel multimodal feature fusion approach is proposed. Methods: In the proposed approach, significant multimodal features are selected respectively by two comparative feature selection methods: Fisher Criterion Score and Davies-Bouldin index. Emotion recognition is performed on the valence-arousal emotion space by using hidden Markov models (HMMs) and multimodal feature sets. Four physiological modalities, including electroencephalogram (EEG) from central nervous system and peripheral physiological signals (PERI) from peripheral nervous system as shown in the DEAP database, are employed. Results: We show the best recognition accuracies of 85.63% for arousal and 83.98% for valence. The proposed feature fusion approach is compared with decision-level fusion and non-fusion approaches on the same database; and the comparison demonstrates significant improvements in accuracy obtained by the feature fusion approach. Conclusion: Our work supports the observation that the proposed feature-level fusion approach represents a promising methodology for emotion recognition.

High-dimensional biomedical datasets contain thousands of features used in molecular disease diagnosis, however many irrelevant or weak correlation features influence the predictive accuracy. Feature selection algorithms enable classification techniques to accurately identify patterns in the features and find a feature subset from an original set of features without reducing the predictive classification accuracy while reducing the computational overhead in data mining. In this paper we present an improved shuffled frog leaping algorithm (ISFLA) which explores the space of possible subsets to obtain the set of features that maximizes the predictive accuracy and minimizes irrelevant features in high-dimensional biomedical data. Evaluation employs the K-nearest neighbour approach and a comparative analysis with a genetic algorithm, particle swarm optimization and the shuffled frog leaping algorithm shows that our improved algorithm achieves improvements in the identification of relevant subsets and in classification accuracy.

Many studies were proposed to discuss acoustic correlates of stress in recent years. Considering some inconsistent experiment results, we supposed that stress should be categorized into long-term and short-term stress in this topic, and the trend of short-term stress induced by workload may be affected by long-term stress. This study contains three parts: first, we proposed an acoustic feature set chosen by feature selection, which can be considered as a measurement of the level of long-term stress; second, we showed that this set is immune to short-term stress in stress classification tests; finally, we observed some particular voice features mentioned in previous researches in our experiment and the results may imply that short-term stress trend is in connection with the level of long-term stress. In short, long-term and shot-term stress should be discussed separately in future researches for clear and explicit conclusions.

In computer-aided diagnosis, a Bayesian classifier that can give the class membership probabilities should be more favorable than classifiers that only give a class assertion. In Bayesian classification, an important and critical step is the probability distribution estimation for each class. Existing methods usually estimate the probability distribution in the whole sample space where the original distribution may be too complex to model. In this paper, we propose a probability distribution estimation method based on local probabilistic model assumption. In our method, the estimation of global probability for a certain point is transformed to the computation of local distribution in a small region, where the local distribution is supposed to be simpler and can be assumed as a simpler probabilistic model. By this method, we implement the Bayesian classifiers based on several local probabilistic model assumptions, and experiments with these classifier have been conducted on several real-word biological and medical datasets; the experimental results demonstrate the efficacy of the proposed method for probabilistic classification in medical diagnosis.

In this paper we propose a biometric solution for individual identification based on electroencephalography with classification using local probability centers. In our study, the electroencephalography signals of a subject are recorded from only one active channel Cz with eyes closed and without any external stimulations. The original signals are preprocessed by Haar wavelet transformation; then a number of features are extracted from the preprocessed signals; and then a classifier with local probability centers are employed to assign the signals to the right person according to the features extracted. By this method we have achieved an average identification accuracy of 96.21% for a dataset of 11 subjects' electroencephalography patterns and the high F-measure values of different persons has shown that this method performed robustly and effectively for various subjects. In addition, we have studied the variation of recognition accuracy with the time length of electroencephalography sessions and found that a longer electroencephalography session is usually more effective for individual identification. These results are in agreement to the previous research and show the evidence that the electroencephalography carries identity information and a longer electroencephalography session usually carries more identity information. With the simple implementation and good performance, we consider our proposed approach to be suitable for development and implementation in a ‘unimodal’ biometric identification system or may be combined with other biometric methods to form a ‘multimodal’ biometric identification system.

The k-nearest neighbor method generates predictions for a particular instance from its neighborhood. It is a simple but effective supervised method for classification. However, the traditional k-nearest neighbor algorithm using the majority voting rule for the class label usually loses a part of useful information in the neighborhood. This paper tries to learn from the neighborhood for more useful information for classification and proposes an improved version of k-nearest neighbor method by heuristically organizing the local distribution characteristics. Different from the traditional methods, the proposed method considers the neighborhood of a query sample from the perspective of local distribution and learns from the neighborhood for local distribution characteristics for classification. We analyze the impact of local distribution characteristics on classification and heuristically develop a formulation to estimate the membership degree, which indicates the level of membership of a query sample to each class; then the query sample is classified to the class which has the highest membership degree with respect to the query sample. Experiments have been conducted on several real data sets; the results support the conclusion that the proposed method is superior to the traditional voting k-nearest neighbor method and comparable with or better than several state-of-the-art methods in terms of classification performance and robustness.

Emotion is an important indicator of depressive conditions. Emotion recognition based on physiological signals such as electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) has gained significant attraction in healthcare domain research. Sharing of physiological signal data related to emotional response between different healthcare systems has the potential to benefit both laboratory-based healthcare research and `real-world' clinical practice. However, management and distribution of the data presents significant challenges; addressing these challenges requires advanced tools for data representation, mining and integration. In this paper we propose such a tool which contains an ontology model called EmotionO+ and rules set based on EEG, which is obtained by random forest algorithm to predict emotional state. It presents not only an effective method to enable semantic representation of the EEG and fNIRS data, but also an emotion knowledge mining tool. Results using EEG data in the eNTERFACE'06 dataset show an accuracy for our proposed model of 99.11% as compared to 97.8% for competing methods using the C4.5 algorithm. The experimental results demonstrate that the posited approach is potentially usable for early stage prediction and intervention for depressive disorders.

Recently, many researchers have used graph theory to study the aberrant brain functions in mental disorders. However, the characteristics of the brain functional network in attention deficit hyperactivity disorder (ADHD) are still largely unexplored. In this study, blood oxygen level-dependence (BOLD) functional magnetic resonance images (fMRI) were employed to construct brain functional networks in 57 children with ADHD and 59 healthy controls (HC). The results showed that both groups had similar global efficiency and the ADHD group had significantly decreased local efficiency compared with the HC. The between-group differences of degree values were found in the frontal cortex, temporal cortex and the default mode network (DMN). The ADHD group retained most of the hub regions found in the HC, but showed altered hub regions in the left posterior cingulate gyrus and the right lingual gyrus. This altered topological organization of the functional network might be associated with the underlying pathophysiology in ADHD.

Recent research has shown that the improvement of mean retrieval effectiveness (e.g., MAP) may sacrifice the retrieval stability across queries, implying a tradeoff between effectiveness and stability. The evaluation of both effectiveness and stability are often based on a baseline model, which could be weak or biased. In addition, the effectiveness-stability tradeoff has not been systematically or quantitatively evaluated over TREC participated systems. The above two problems, to some extent, limit our awareness of such tradeoff and its impact on developing future IR models. In this paper, motivated by a recently proposed bias-variance based evaluation, we adopt a strong and unbiased "baseline", which is a virtual target model constructed by the best performance (for each query) among all the participated systems in a retrieval task. We also propose generalized bias-variance metrics, based on which a systematic and quantitative evaluation of the effectiveness-stability tradeoff is carried out over the participated systems in the TREC Ad-hoc Track (1993-1999) and Web Track (2010-2012). We observe a clear effectiveness-stability tradeoff, with a trend of becoming more obvious in more recent years. This implies that when we pursue more effective IR systems over years, the stability has become problematic and could have been largely overlooked.

Modeling and prediction of Electroencephalogram (EEG) signals is very important for Portable applications; EEG signals are however widely regarded as being chaotic in nature. An adaptive modeling technique that combines Discrete Wavelet Transformation (DWT) to predict contaminated EEG signals for removal of ocular artifacts (OAs) from EEG records is proposed as an effective a data processing tool for Interventions in Mental Illness Based on Bio-feedback. The proposed method is well suited for use in portable environments where constraints with respect to acceptable wearable sensor attachments usually dictate single channel devices. Using simulated and measured data the accuracy of the proposed model is compared to the accuracy of other pre-existing methods based on Wavelet Packet Transform (WPT) and independent component analysis (ICA) using DWT and adaptive noise cancellation (ANC) for Portable applications. The results show that the our new model not only demonstrates an improved performance with respect to the recovery of true EEG signals, achieves improved computational speed, and demonstrates better tracking performance.

With the development of the electronic health record, sharing biomedical and healthcare data among heterogeneous systems has great potential to benefit both clinical healthcare and scientific research. Standardizing the data presentation is the best solution to exchange them among heterogeneous systems. However, electroencephalogram (EEG) data has met problems since existing diverse data formats are unsuitable for this purpose. To address this problem we introduce an XML-based data presentation method named eegML for EEG in this paper. Based on advantages of XML technologies, eegML is a generic system- and application-independent and flexible open solution to the effective presentation of EEG data. Besides, eegML could also be used to aid knowledge management and discovery in biomedical and healthcare informatics. This format, if widely adopted, could better promote EEG data management and mining.

Accounting for a patient's emotional state is integral in medical care. Positive emotions have a significant influence on mental and physical health. Much research has been carried out on the impact of emotion on the development and course of different illnesses. Emotion and significant contexts have an important role in helping people cope with depression, but lack of support undermines coping. In this paper, we present an emotion-focused e-health monitoring support system in assisting them to cope with their illness. In the system proposed, a user could monitor his emotional states in the absence of a doctor and regulate his emotions through the system. To provide personalized health care services to the user anywhere and anytime, the system should convert low-level multimodal context (including physiological signal, user profile and environment information) to high-level context. The objective of this research is establishing an emotion-focused e-health system for health care services.

In an increasingly knowledge based world, people are confronted with an explosion of information from the environment which must be viewed in restricted attention spans. Hence there is a need to investigate how best to model our Visual Attention (VA) with a view to allocate our attention efficiently. We use the color-word Stroop task combined with electroencephalogram (EEG) to model VA: subjects undertake the Stroop task and their EEG is recorded. This is in contrast to other studies that use techniques such as Event Related Potentials (ERP), Contextual Modeling Frameworks, eye movements and facial recognition. The paper presents a simple and useful model to recognize VA dynamically. We use the linear EEG features of different cortical fields as the main inference factors, and take the response time (RT) of the Stroop task as a metric to quantify subject performance. First, we obtain the most relevant EEG feature vectors from the recording, using a correlation analysis. Second, we use experimental data for training the VA model, using a regression method. Last, we then apply further experimental data to test the proposed model. The results from the tests conducted demonstrate that our model maps visual attention very closely.

In this study, the amplitude of low frequency (0.01-0.08Hz) fluctuation (ALFF) and functional connections were used to analyze blood oxygen level-dependent (BOLD) functional magnetic resonance imaging (fMRI) data for 57 children with attention deficit hyperactivity disorder (ADHD) and 59 healthy controls (HC) in a resting state. Compared with HC, the ADHD showed significantly altered neural activity in the prefrontal and temporal cortex. In ADHD, altered functional connections were observed mainly in prefrontal cortex, temporal cortex, default mode network (DMN) and parahippocampal gyrus (PHG). The abnormalities in prefrontal and temporal cortex might provide evidence to support that functional abnormalities are extended from the prefrontal cortex to the temporal cortex in ADHD. Additionally, dysfunction of the DMN and PHG could be related to the pathophysiological mechanisms of ADHD.

A new model to remove ocular artifacts (OA) from electroencephalograms (EEGs) is presented. The model is based on discrete wavelet transformation (DWT) and adaptive noise cancellation (ANC). Using simulated and measured data, the accuracy of the model is compared with the accuracy of other existing methods based on stationary wavelet transforms and our previous work based on wavelet packet transform and independent component analysis. A particularly novel feature of the new model is the use of DWTs to construct an OA reference signal, using the three lowest frequency wavelet coefficients of the EEGs. The results show that the new model demonstrates an improved performance with respect to the recovery of true EEG signals and also has a better tracking performance. Because the new model requires only single channel sources, it is well suited for use in portable environments where constraints with respect to acceptable wearable sensor attachments usually dictate single channel devices. The model is also applied and evaluated against data recorded within the EUFP 7 Project--Online Predictive Tools for Intervention in Mental Illness (OPTIMI). The results show that the proposed model is effective in removing OAs and meets the requirements of portable systems used for patient monitoring as typified by the OPTIMI project.

With the booms of mobile communication, especially mobile smart phone, technologies to identify individuals for mobile security calls for some more strict requirements in user-friendly, real-time and ubiquitous aspects. In addition to traditional approaches (for example, password check), some advanced biometric methodologies have been applied in practice, such as fingerprint and iris based solutions, however, these solutions generally lack a true ubiquitous nature for mobile security. In this paper, we present a real time EEG based individual identification interface to support ubiquitous applications. The EEG signals are collected through a mono-polar single channel in real time via a mobile EEG device. An experiment involving about 20 subjects has been conducted to evaluate the interface. The experiment comprises three types of tests: accuracy test, time dimension test and capacity dimension test. The results of these experiments demonstrate that our approach is highly suitable to the demands of mobile security in ubiquitous environment. In addition, we integrate this interface into scenarios of ubiquitous application - Online Predictive Tools for Intervention in Mental Illness (OPTIMI).

Rule-based systems have been shown to enable automated problem solving and decision-support. Rule-based systems are however inherently domain specific requiring strategic rules design to address the requirements specification for the problem under consideration. Rule-based systems fall into two (general) types: traditional and fuzzy rule-based systems. This paper considers rule types and strategies with an introduction to logic systems as they apply to the approach proposed in this paper and an overview of related research. Consideration is given to the principles that underlie rule-based system design. The novel Context Processing Rules and Fuzzy Event Condition Action rules are presented with an evaluation and proof-of-concept. This paper posits that fuzzy rule-based systems provide an effective approach to enable context processing with constraint satisfaction in pervasive intelligent context-aware systems with decision-support.

The Ubiquitous Awareness and Intelligent Solutions Lab (UAIS) was established in January 2009, at the School of Information Science and Engineering, Lanzhou University, China. It mainly focuses on research in pervasive computing, affective learning, the semantic web and CSCW. The lab encourages multidisciplinary cooperation among its researchers who have backgrounds in computer science, medicine, mathematics and signal processing. UAIS has undertaken research projects sponsored by the National Science Foundation China, European Framework Programme (FP7) and "985" and "211" projects from Lanzhou University. UAIS has also established strong connections with research institutes at home and abroad as well as with companies to facilitate its research and training of students. Copyright 2011 ACM.

The goal of the workshop is to build a forum for researchers from academy and industry to investigate challenging and innovative research issues in the subject, which combines Affective Interaction within ubiquitous environment and to explore creative concepts, theories, innovative technologies and intelligent solutions. Potential participants may come from communities of ubiquitous computing, intelligent computing, brain computer interaction, affective computing, neuroergonomics, cognitive neuroscience etc. in order to present their state-of-the-art progress and visions on the various overlaps across those disciplines. In this proposal, we describe the detailed purpose, topics, and format of this workshop on "Ubiquitous Affective Awareness and Intelligent Interaction".

Context has been applied to a broad range of domains and systems ranging from location-based tourist applications to motion capture. An area relatively under-represented in the literature is health-monitoring systems. This paper considers the application of context in health monitoring and the role of translational research, a concept that relates to the translation of basic research into clinical practice. This paper posits that intelligent Context-aware systems provide an effective basis for translational research with benefits reflected in improved quality of life for patients, reduced load on clinicians, and efficiency improvements. Case studies are presented to illustrate a number of health monitoring issues and how context can be effectively applied. The paper concludes with a discussion, consideration of practical applications, challenges, and open research questions.

Context-aware systems provide an effective basis upon which the processing of captured and user-entered data (contextual information) with a direct relationship to an individuals current personal and environmental situation (termed `situated role') can be achieved. Context-aware systems have significant potential benefits for the individual, academia, and industry alike, the use of context being evident in a broad and diverse range of domains and systems. The implementation of context however introduces the issue of constraint satisfaction and preference compliance. This paper considers this issue and presents an algorithm incorporating weighting applied to context properties in a quantitative approach to enable mitigation of violations of constraints and expressed preferences. The algorithm is presented with proof-of-concept. Implementation is considered with related issues and projected future research.

Bo Hu, Bin Hu, Victor Callaghan, Zongkai Lin; Combining Theory and Systems Building in Pervasive Computing, The Computer Journal, Volume 53, Issue 2, 1 February

This paper addresses context in intelligent context-aware systems to support personalised service provision and cooperative computing. Context processing, context modelling, ontology, and OWL are introduced and a context reasoning ontology presented. Context implementation reduces to a decision problem which is characterised as one of selecting from a number of potential options based on the relationship between the values that describe the input and the solution, the modelling school of decision analysis attempts to construct an explicit model of such relationships, usually in the form of decision trees. An overview of decision trees with parametric design considerations is presented. Comparisons with related research are drawn and an evaluation and simulation of Smart-Context is presented. RDF/S with OWL and Jena provide an effective basis for autonomous decision making using processing rules, and the issue is one of implementation in adaptable and tractable solutions. A conclusion with open research questions is presented with consideration of potential directions for future research.

By Bin Hu, Guest Editor Pervasive Computing envisions a world of communicating small and potentially portable devices, sensors, and actuators. Actually, it is a rapidly developing area of Information and Communications Technology. The term refers to the increasing integration of Information and Communications Technology (ICT) into people's life and environments, made possible by the growing availability of microprocessors with inbuilt communication facilities. Pervasive computing has many potential applications, from health and home care to environment monitoring and intelligent transport systems. This briefing provides an overview of pervasive computing technologies and discusses the growing debate over privacy, safety, and environmental implications. Among the myriad of tools expected to support there are application-specific integrated circuitry; speech recognition; gesture recognition; system on a chip; perceptive interfaces; smart matter; flexible transistors; reconfigurable processors; field programmable logic gates; and microelectromechanical systems. This special issue showcases papers that describe recent research in these domains. Let the contents of the articles be briefly summarized. In “A novel task scheduling algorithm based on dynamic critical path and effective duplication for pervasive computing environment,” Junzhou Luo, Fang Dong, Jiuxi Cao, and Aibo Song propose a novel task scheduling algorithm based on dynamic critical path (DCP) and effective duplication, called DCPED, in which a more accurate DCP calculation method that takes resource availability into account is introduced. Mental health care is a major cost to all European Union (EU) nations. Although cognitive behavior therapy (CBT) is an effective treatment of choice for mental disorders, it still suffers from a lack of accessibility and personalization. Bin Hu, Bo Hu, Jizheng Wan, Majoe Dennis, Hsiao-Hwa Chen, Lian Li, and Qingguo Zhou in their contribution “Ontology-based ubiquitous monitoring and treatment against depression” attempt to propose an ontology-enhanced ubiquitous monitoring and treatment model for the purpose of assisting people to overcome the challenges of mental disorders. In “A novel portable multimedia QoS monitor: independent and high efficiency” by Qingguo Zhou, Shuwei Bai, Bin Hu, Nicholas MC Guire, and Lian Li, the main function of their novel multimedia QoS monitor, portable multimedia monitor (p3m), is data package analysis and statistics. The system adopts multimedia packages management (MMPM) mechanism which can make monitor tool become server-independent and monitor the multimedia packages by preference. The interaction modality constrained to desktop PC between users and workflow management system is one of the main factors obstructing the wide application of business process in pervasive computing environments. The paper “Business process management with group features in pervasive environment” of Wei He, and Haiyang Wang introduces a new modeling method for multifold group activity as well as the mechanism supporting interaction between users and systems with multiple modalities. In “Towards life long learning: three models for ubiquitous applications”, Andreas Holzinger, Alexander Nischelwitzer, Silvia Friedl, and Bo Hu present three experimental proof-of-concepts, the implementation of which can support life long learning, when one follows the working-is-learning paradigm. In their paper “Generic service composition platform for pervasive E-Commerce,” Jing Chi, Chunyang Yin, Meina Song, Junde Song, and Xiaosu Zhan present a generic E-Commerce process model with full E-Commerce service coverage, based on which a generic service composition platform is proposed as an individual middleware layer to support the pervasive E-Commerce. Bo Liu, Junzhou Luo, and Jiuxin Cao believe that it is an effective approach to adopt software agent in pervasive computing (Per-Com) environment and task execution is the key problem which is affected directly by task representation method in Per-Com environment. They present an agent-based Per-com architecture, an agent role based task relationship tree model, an XML-based agent task representation method and an agent task decomposing, and online multi-agent scheduling algorithm in their paper “Agent-based task representation and processing in pervasive computing environment.” The guest editor is convinced that all papers selected for this special issue constitute valuable contributions to many different areas in Pervasive Computing and related fields. He thanks to the 30 referees who served with their rigorous referee reports and to all colleagues who participated in this exciting endeavor with care and vision for their highly appreciated help.

Abstract Mental health care is a major cost to all EU nations, and in many case, it results in additional costs to a country's economy due to the lost of productivity and concerns at the societal level. Cognitive behaviour therapy (CBT) is by far the treatment of choice for mental disorders such as depression, anxiety disorders, eating disorders, chronic insomnia, etc. Although effective, CBT suffers from a lack of accessibility and personalisation. In this paper, we proposed an ontology‐enhanced ubiquitous monitoring and treatment model for the purpose of assisting people to overcome the challenges of mental disorders. We first presented a context ontology for mental disorders as the basis upon which semantics‐enhanced methods are developed for gathering, formalising and manipulating patients' data. We implemented the proposed framework to facilitate online CBT, for treating depression at the current stage, which combines together talk/chat/messaging services, helps in retrieving neurofeedback, and supports collaborative diagnosis when necessary. An online statistic report was also integrated into our system. Finally, the paper discussed the proposed framework by comparing it against relevant research in the field and elaborated on possible further research directions. Copyright © 2009 John Wiley & Sons, Ltd.

Abstract In this paper, a novel multimedia QoS monitor, p3m (Portable MultiMedia Monitor), is presented. The main function of p3m is data package analysis and statistics. Breaking through the conventional method, this new QoS monitor moves jobs into kernel space, by which to meet the real‐time requirements of multimedia services. With the development of broadband Internet, more and more heterogeneous multimedia services and protocols have been widely deployed. As known, there are two traditional QoS monitors. One is server‐based, depending on the specific multimedia server, such as Helix. Another is tcpdump‐based, e.g. mmdump, which can measure the nature of multimedia traffic merely. However, it is still difficult to monitor the QoS of multimedia networks by these two traditional strategies. To resolve the problems, our new system adopts MMPM (MultiMedia Packages Management) mechanism which can make monitor tool become server‐independent and monitor the multimedia packets by preference, thus greatly reducing resource requirements with offering high performance. This paper describes the design and implementation of p3m and demonstrates one instance of its utility in monitoring the QoS of RTSP with Helix Server. The evaluation of p3m is also presented in this paper. Copyright © 2009 John Wiley & Sons, Ltd.

Developments in higher education have driven interest in personalised education. Concomitant with these developments are the evolving capabilities of mobile technologies. Context has been shown to facilitate personalisation in mobile systems, the issue is how to effectively create and implement an individual's profile (termed context). dasiaIntelligent Context' provides a basis for realising an effective solution to this issue. This paper briefly considers context, introduces 'Intelligent Context' and presents a brief evaluation. The paper concludes with conclusions and future research.

BACKGROUND: Identification of protein complexes is crucial for understanding principles of cellular organization and functions. As the size of protein-protein interaction set increases, a general trend is to represent the interactions as a network and to develop effective algorithms to detect significant complexes in such networks. RESULTS: Based on the study of known complexes in protein networks, this paper proposes a new topological structure for protein complexes, which is a combination of subgraph diameter (or average vertex distance) and subgraph density. Following the approach of that of the previously proposed clustering algorithm DPClus which expands clusters starting from seeded vertices, we present a clustering algorithm IPCA based on the new topological structure for identifying complexes in large protein interaction networks. The algorithm IPCA is applied to the protein interaction network of Sacchromyces cerevisiae and identifies many well known complexes. Experimental results show that the algorithm IPCA recalls more known complexes than previously proposed clustering algorithms, including DPClus, CFinder, LCMA, MCODE, RNSC and STM. CONCLUSION: The proposed algorithm based on the new topological structure makes it possible to identify dense subgraphs in protein interaction networks, many of which correspond to known protein complexes. The algorithm is robust to the known high rate of false positives and false negatives in data from high-throughout interaction techniques. The program is available at http://netlab.csu.edu.cn/bioinformatics/limin/IPCA.

This paper addresses context issues in pervasive mobile computing and presents a semantic context model to support location based cooperative mobile applications. In this paper, a mobile learning prototype has been discussed as a scenario for evaluation of the context model. A Bluetooth based wireless solution is introduced for the purpose of integrating this model with the current e-learning system at UCE Birmingham. The aim of our research is to investigate some essential issues on a suitable context pattern applicable to contextual cooperative mobile computing.

This paper presents a context model to support location based cooperative mobile learning. In the model, a standardised context template with a context middleware has been introduced to investigate some essential issues on a suitable context pattern applicable to cooperative mobile learning. Its integration with the current e-learning system in University of Central England is also included in this paper

This paper presents the development of an RDF context framework to enable the identification of an individual student in the domain of higher education to facilitate targeted resource delivery and the matching of users in collaborative learning. The framework is designed to be extensible and generic with the ability to generalise to similar domains. Context definitions are domain specific, each domain requiring specific contextual design and the identification of factors that identify an entity in a specific domain. The factors that identify a student context with the developed RDF context framework are presented in this paper