Self-supervised learning advances audio representation for multimedia analysis. However, prevailing data-centric approaches rely on massive real-world corpora, increasing training costs, curation burdens, and privacy barriers. To address this, we present AudioPG, a procedural synthesis framework eliminating real audio recordings during pre-training. AudioPG trains a Transformer-based masked autoencoder on waveforms generated on-the-fly from basic acoustic primitives and composition rules. The encoder transfers effectively to real audio benchmarks, achieving 90.60% accuracy on ESC-50, 0.546 mAP on FSD50K, 88.17% on UrbanSound8K, and 97.03% on Speech Commands V2. Notably, pre-training completes in under 20 minutes on a single GPU. Latent space analysis reveals physical factors, including fundamental frequency and relative intensity, emerge in orthogonal subspaces, making representations linearly decodable. These results establish procedural synthesis as an efficient, interpretable pre-training signal when large-scale corpora are unavailable. Our code is available at: https://github.com/Freyliu0516/audioPG.

Existing manual labeling of micro-expressions is subject to errors in accuracy, especially in cross-cultural scenarios where deviation in labeling of key frames is more prominent. To address this issue, this paper presents a novel Global Anti-Monotonic Differential Selection Strategy (GAMDSS) architecture for enhancing the effectiveness of spatio-temporal modeling of micro-expressions through keyframe re-selection. Specifically, the method identifies Onset and Apex frames, which are characterized by significant micro-expression variation, from complete micro-expression action sequences via a dynamic frame reselection mechanism. It then uses these to determine Offset frames and construct a rich spatio-temporal dynamic representation. A two-branch structure with shared parameters is then used to efficiently extract spatio-temporal features. Extensive experiments are conducted on seven widely recognized micro-expression datasets. The results demonstrate that GAMDSS effectively reduces subjective errors caused by human factors in multicultural datasets such as SAMM and 4DME. Furthermore, quantitative analyses confirm that offset-frame annotations in multicultural datasets are more uncertain, providing theoretical justification for standardizing micro-expression annotations. These findings directly support our argument for reconsidering the validity and generalizability of dataset annotation paradigms. Notably, this design can be integrated into existing models without increasing the number of parameters, offering a new approach to enhancing micro-expression recognition performance.

Multi-modal data integration and missing data handling present significant challenges across various domains, particularly in stock market prediction and arrhythmia detection. Existing approaches frequently fail to capture the complex dependencies and heterogeneous dynamics in multi-modal data, resulting in suboptimal performance. To address these limitations, we propose MTLP-MDG: a novel Multi-Task Learning framework with Probabilistic Distribution Perception for Missing Data Generation. Our framework comprises three innovative components: (1) a Probabilistic Distribution Perception module (PDP-MDG) that leverages Deep Belief Networks to learn underlying data distributions and generate missing values with statistical fidelity; (2) an Alternative Telescopic Displacement (ATD) Fusion module that adaptively integrates heterogeneous data modalities through iterative scaling, rotation, and displacement operations; and (3) a Multi-Task Learning Framework (MTLP) that optimizes shared representations across related tasks while maintaining task-specific objectives. Comprehensive evaluations on the S&P500 dataset for stock market prediction and the MIT-BIH Arrhythmia Database demonstrate that our framework significantly outperforms state-of-the-art baselines, achieving superior performance in stock price movement prediction, classification, and arrhythmia detection. The MTLP-MDG framework provides an effective solution for multi-modal data integration and missing data handling, with demonstrated applicability across diverse domains.

Multimodal learning faces two key challenges: effectively fusing complex information from different modalities, and designing efficient mechanisms for cross-modal interactions. Inspired by neural plasticity and information processing principles in the human brain, this paper proposes BC-PMJRS, a Brain Computing-inspired Predefined Multimodal Joint Representation Spaces method to enhance cross-modal learning. The method learns the joint representation space through two complementary optimization objectives: (1) minimizing mutual information between representations of different modalities to reduce redundancy and (2) maximizing mutual information between joint representations and sentiment labels to improve task-specific discrimination. These objectives are balanced dynamically using an adaptive optimization strategy inspired by long-term potentiation (LTP) and long-term depression (LTD) mechanisms. Furthermore, we significantly reduce the computational complexity of modal interactions by leveraging a global-local cross-modal interaction mechanism, analogous to selective attention in the brain. Experimental results on the IEMOCAP, MOSI, and MOSEI datasets demonstrate that BC-PMJRS outperforms state-of-the-art models in both complete and incomplete modality settings, achieving up to a 1.9% improvement in weighted-F1 on IEMOCAP, a 2.8% gain in 7-class accuracy on MOSI, and a 2.9% increase in 7-class accuracy on MOSEI. These substantial improvements across multiple datasets demonstrate that incorporating brain-inspired mechanisms, particularly the dynamic balance of information redundancy and task relevance through neural plasticity principles, effectively enhances multimodal learning. This work bridges neuroscience principles with multimodal machine learning, offering new insights for developing more effective and biologically plausible models.

Emotion recognition in conversation has attained increasing attention for perceiving user emotion in practical conversational applications. Conversational utterances spoken alternately by different speakers inspire most studies to leverage speaker information based on golden speaker labels. In this work, we challenge the existing paradigm of utilizing available speaker labels with a more realistic scenario, where the speaker identity of each utterance is unknown during inference. We propose Progressive Contrastive Deep Supervision for multimodal emotion recognition in conversation (PCDS), incorporating speaker diarization and emotion recognition into one unified framework. To facilitate joint task learning, we inject speaker and emotion bias into the network progressively via contrastive deep supervision, with the task-irrelevant contrast being the intermediate transition. To obtain explicit speaker dependency, we propose a speaker contrast and clustering module (SCC) to endow the capability of partitioning speakers into groups even when neither speaker label nor number of speakers is known as a priori. Experiments on two ERC benchmarks, including IEMOCAP and MELD demonstrate the effectiveness of the proposed method. We also show that progressive contrastive deep supervision helps reconcile the underlying tension between speaker diarization and emotion recognition. Source code is available from Github[https://github.com/Cross-Innovation-Lab/PCDS/].

Multimodal sentiment analysis (MSA) has emerged as a prominent research area in the field of computer science, focusing on the comprehension of human behaviors by computational systems. Most existing pipelines usually involve two steps: unimodal representation and multimodal fusion. However, on one hand, within this process, the varying contributions of different modalities can cause imbalance in multimodal training. On the other hand, in unimodal representation, the inclusion of pretrained models presents challenges of slow training and difficult optimization. Based on the aforementioned findings, we have redefined the workflow for current multimodal emotion recognition. In this study, we introduce a novel approach called reward-based gradient modulation for regulating the convergence speed of individual modalities within the fusion network with LoRA (RGM-LoRA), aimed at achieving a balanced integration process. Additionally, text modality encoders commonly employ large-scale language pretraining models such as BERT. We introduce LoRA to alleviate the problem of text modality optimization being suppressed by the other two modalities. To our knowledge, we are the pioneering researchers who have provided evidence showcasing the efficacy of LoRA in the context of optimization. Finally, to further ensure the effect of textual modality, we add intermodal contrast learning. As a result, we achieve the state-of-the-art (SOTA) performance on two public benchmark datasets, CMU-MOSI, and CMU-MOSEL.

The mainstream paradigm of speech emotion recognition (SER) is identifying the single emotion label of the entire utterance. This line of works neglect the emotion dynamics at fine temporal granularity and mostly fail to leverage linguistic information of speech signal explicitly. In this paper, we propose Emotion Neural Transducer for fine-grained speech emotion recognition with automatic speech recognition (ASR) joint training. We first extend typical neural transducer with emotion joint network to construct emotion lattice for fine-grained SER. Then we propose lattice max pooling on the alignment lattice to facilitate distinguishing emotional and non-emotional frames. To adapt fine-grained SER to transducer inference manner, we further make blank, the special symbol of ASR, serve as underlying emotion indicator as well, yielding Factorized Emotion Neural Transducer. For typical utterance-level SER, our ENT models outperform state-of-the-art methods on IEMOCAP in low word error rate. Experiments on IEMOCAP and the latest speech emotion diarization dataset ZED also demonstrate the superiority of fine-grained emotion modeling. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ENT.

Change address identification is one of the difficulties in bitcoin address clustering as an emerging social computing problem. Most of the current-related research only applies to certain specific types of transactions and faces the problems of low recognition rate and high false positive rate. We innovatively propose a clustering method based on multiconditional recognition of one-time change addresses and conduct experiments with on-chain bitcoin transaction data. The results show that the proposed method identifies at least 12.3% more one-time change addresses than other heuristics. On top of the multi-input heuristic clustering method, the proposed method also improves the address clustering performance by 5.7%, achieves optimal recognition results compared with similar methods, and significantly reduces the false positive rate of recognition results. This work provides the technical basis for antimoney laundering efforts based on entity identification. Code and data could be accessed from https://github.com/ECNU-Cross-Innovation-Lab/BitcoinAddressClustering.

Dynamic Facial Expression Recognition (DFER) is a rapidly developing field that focuses on recognizing facial expressions in video format. Previous research has considered non-target frames as noisy frames, but we propose that it should be treated as a weakly supervised problem. We also identify the imbalance of short- and long-term temporal relationships in DFER. Therefore, we introduce the Multi-3D Dynamic Facial Expression Learning (M3DFEL) framework, which utilizes Multi-Instance Learning (MIL) to handle inexact labels. M3DFEL generates 3D-instances to model the strong short-term temporal relationship and utilizes 3DCNNs for feature extraction. The Dynamic Long-term Instance Aggregation Module (DLIAM) is then utilized to learn the long-term temporal relationships and dynamically aggregate the instances. Our experiments on DFEW and FERV39K datasets show that M3DFEL outperforms existing state-of-the-art approaches with a vanilla R3D18 backbone. The source code is available at https://github.com/faceeyes/M3DFEL.

Fueled by recent advances of self-supervised models, pre-trained speech representations proved effective for the downstream speech emotion recognition (SER) task. Most prior works mainly focus on exploiting pre-trained representations and just adopt a linear head on top of the pre-trained model, neglecting the design of the downstream network. In this paper, we propose a temporal shift module to mingle channel-wise information without introducing any parameter or FLOP. With the temporal shift module, three designed baseline building blocks evolve into corresponding shift variants, i.e. ShiftCNN, ShiftLSTM, and Shiftformer. Moreover, to balance the trade-off between mingling and misalignment, we propose two technical strategies, placement of shift and proportion of shift. The family of temporal shift models all outperforms the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER.

Dynamic graph learning has received increasing attention in recent years. However, real-world graph data sets are characterized by significant structural complexity, attribute diversity, and temporal variability. Importantly, there are complex and significant influence mechanisms between them. All them pose great challenges to dynamic graph learning (DGL). To address them, we propose a novel dynamic graph learning framework, MTT-DynGL. First, graph attention networks (GAT) is used to efficiently aggregate the topology and multidimensional attribute features on each snapshot. Then, a temporal variation matrix with strength factors is designed to further measure the interaction mechanism between structures and attributes over time. Further, to effectively integrate the above results, a MTT-based dynamic graph learning network is designed. It consists of an MTT integration mechanism and a bidirectional dilated causal convolution network. The former is used to learn temporal variation features in an integrated manner, and the latter is used to improve learning quality and training efficiency. Finally, the effectiveness of our method is verified by multiple experiments.

In recent years, it is a difficult issue to integrate the deep cross-fertilization and interpretable cognitive modeling methods from the basic theory of emotional psychology with deep learning and other algorithms. To address this problem, a cognitive model that integrates the VGG-facial action coding system (FACS)-OCC model based on fer2013 expression features and the OCC-pleasure-arousal-dominance (PAD)-openness, conscientiousness, extraversion, agreeableness, and neuroticism (OCEAN) fusion of the basic theory of emotional psychology, namely, a computational affection-based OCC-PAD-OCEAN federation cognitive modeling (OPO-FCM), is constructed. By constructing this model and performing formal proof algorithms, it is shown that the OPO-FCM can acquire expression features in video streams, complete the acquisition of expression features in videos by training a deep neural network, map expressions to the PAD emotion space through the established expression–basic emotions–emotion space mapping relationship, and finally complete the mapping of the average emotion over a period time. The information of personality space is obtained through it. Finally, the experimental simulation of the model is conducted, and the results show that the average accuracy of the valid tested personalities is 79.56%. This article takes the knowledge-driven approach of emotional psychology as a starting point and combines deep learning techniques to construct interpretable cognitive models, thus providing new ideas for future cross-innovation between computer technology and psychology theory.

Fusion technology is crucial for multimodal sentiment analysis. Recent attention-based fusion methods demonstrate high performance and strong robustness. However, these approaches ignore the difference in information density among the three modalities, i.e., visual and audio have low-level signal features and conversely text has high-level semantic features. To this end, we propose a non-homogeneous fusion network (NHFNet) to achieve multimodal information interaction. Specifically, a fusion module with attention aggregation is designed to handle the fusion of visual and audio modalities to enhance them to high-level semantic features. Then, cross-modal attention is used to achieve information reinforcement of text modality and audio-visual fusion. NHFNet compensates for the differences in information density of different modalities enabling their fair interaction. To verify the effectiveness of the proposed method, we set up the aligned and unaligned experiments on the CMU-MOSEI dataset, respectively. The experimental results show that the proposed method outperforms the state-of-the-art. Codes are available at https://github.com/skeletonNN/NHFNet.

Abstract Learning unimodal representations and improving multimodal fusion are two cores of multimodal sentiment analysis (MSA). However, previous methods ignore the information differences between different modalities: Text modality has high‐order semantic features than other modalities. In this article, we propose a sparse‐ and cross‐attention (SCANET) framework which has asymmetric architecture to improve performance of multimodal representation and fusion. Specifically, in the unimodal representation stage, we use sparse attention to improve the representation efficiency of two modalities and reduce the low‐order redundant features of audio and visual modalities. In the multimodal fusion stage, we design an innovative asymmetric fusion module, which utilizes audio and visual modality information matrix as weights to strengthen the target text modality. We also introduce contrastive learning to effectively enhance complementary features between modalities. We apply SCANET on the CMU‐MOSI and CMU‐MOSEI datasets, and experimental results show that our proposed method achieves state‐of‐the‐art performance.

Abstract Skeleton‐based action recognition has been extensively studied in recent years and applied in virtual reality, detection systems and other cases with strong requirements for low cost as well as high accuracy, but most of the existing methods mainly focus on complex architecture of deep neural networks without considering computation efficiency. To balance accuracy and computation cost well, this paper proposes a simple and efficient attention‐enhanced semantic‐guided graph convolutional network (AeS‐GCN) for skeleton‐based action recognition. Firstly, we fuse semantics of joint type and frame index and dynamics together as representation of skeleton. Then, we use spatial attention block (SAB) to explore important features in spatial structure, in which adaptive GCN layer is adopted to adaptively model skeleton topology structure. Next, we use temporal attention block (TAB) to extract latent temporal information. The model proposed is a lightweight network and achieves the state‐of‐the‐art performance on mainstream datasets with less parameters and less computational complexity.

Spectrum detection plays an important role in 5G communication and Internet of things (IoT) networks which is one of the core technologies in wireless communication network. The process of spectrum detection largely depends on energy sensing as this process needs to detect the energy of different frequency bands in the information transmission. In this paper, we propose an energy-aware scheme based on spectrum detection in wireless sensor networks applications, which can save the cost of data collection and computing power of DDoS attack de-tection without collecting a large number of traffic characteris-tics and large-scale deep learning model training. By detecting the information transmission energy in different frequency channels, the busy and idle state of the channel is judged. On the basis of the busy channel detection, the scope of DDoS attack is further judged by the topology of wireless sensor networks. Sim-ulation results show that the proposed energy-aware algorithm can achieve the expected efficiency and accuracy, and is suitable for low load and high security requirements of wireless sensor networks.

With the continuous development and popularity of depth cameras, skeleton-based human action recognition has attracted people's wide attention. Graph Convolutional Network (GCN) has achieved remarkable performance. However, the existing methods do not better consider the semantic characteristics, which can help to express the current concept and scene information. Semantic information can also help with better granularity classification. In addition, most of the existing models require a lot of computation. What's more, adaptive GCN can automatically learn the graph structure and consider the connections between joints. In this paper, we propose a relatively less computationally intensive model, which combines semantic and adaptive graph network (SAGN) for skeleton-based human action recognition. Specifically, we mainly combine the dynamic characteristics and bone information to extract the data, taking the correlation between semantics into the model. In the training process, SAGN includes an adaptive network so that we can make attention mechanism more flexible. We design the Convolutional Neural Network (CNN) for feature extraction on the time dimension. The experimental results show that SAGN achieves the state-of-the-art performance on NTU-RGB+D 60 and NTU-RGB+D 120 datasets. SAGN can promote the study of skeleton-based human action recognition. The source code is available at https://github.com/skeletonNN/SAGN.

Cyber-physical system integrates physical entity and its virtual model, which facilitates intelligent manufacturing a lot. Due to the geometric and non-geometric factors, transmission delay between actual and virtual environment, errors exist between desired position and actual position in real-time movement. Thus the accuracy and efficiency need to be improved. In this paper, a mapping approach of the actual UR10 robot and its virtual model based on long short-term memory neural network is developed to implement the synchronization of the virtual-real UR10 twins' behaviors in cyber-physical system. The virtual model can reflect and control the behaviors of UR10 in real time and vice versa. This method is based on a time recursive structure thus takes the temporal property of trajectory points into account. A prototype system is developed to validate its effectiveness. Experimental validation is conducted to compare the LSTM based calibration method with existing kinematic methods and multilayer perceptron neural net based methods. As demonstrated in the experiment results, the real-time mapping model of the virtual-real UR1O twins' behaviors can be obtained.

Cloud computing technologies have enabled a new paradigm for intelligent manufacturing system which is powered by utilizing distributed resources, such as collaborative robots, simulation engines, advanced algorithms and human resources. As one of the key issues, the mechanism for online kinematics control of serial robotic manipulator presents speed challenge in the cloud-based system. In this research, a kinematics approximation model based on kernel ridge regression is developed for cloud manufacturing environment. To begin with, the model input is generated using trigonometric functions of rotation angles with permutation tricks which significantly reduces statistical error. Then, the approximation model is trained using kernel ridge regression with radial basis function, where both regularization and bandwidth of kernel have been optimized using grid-search. In addition, Universal Robot 10 is adapted as a collaborative robot example in simulation comparison experiments in order to evaluate the performance of the kinematics approximation model. As demonstrated in the experiment results, the proposed modelling approach can effectively support the cloud simulation paradigm and efficiently meet the real-time speed requirement in a distributed manufacturing environment.