Micro-expression recognition (MER) is a challenging task due to the subtle and short-lived facial muscle movements involved. Macro-expressions, in contrast, are more evident and easy to recognize. Yet, both expressions share similar facial muscles to express the same emotions. We exploit this observation to propose a novel Macro-expression guidance network (MAG) that uses motion similarity to aid MER. The MAG has four key components: 1) motion vectorization, which transforms facial motion into a vector tensor representation to capture motion dynamics; 2) nonlinear amplification, which enhances the intensity of micro-expression features to make them more salient; 3) macro-micro matching, which aligns macro- and micro-expressions with the highest motion similarity to achieve a one-to-one mapping between the two modalities; and 4) guidance mechanism, which enables macro-expressions to guide the extraction of micro-expression features using convolutional operations. We perform extensive experiments on 7 datasets under 3 benchmarks and demonstrate that MAG outperforms state-of-the-art methods for MER.

Speech represents a fundamental medium for conveying human emotions and, as a result, speech-based emotion recognition (SER) systems have become pivotal in advancing human-computer interaction (HCI) across a range of applications. While significant progress has been made in speech emotion recognition over recent years, existing solutions still face several key challenges, in that they:$(i)$rely excessively on subjectively annotated (discrete) labels during training,$(ii)$often overlook the label ambiguity of speech samples that express more than one class of emotions, and$(iii)$underutilize unlabeled or ambiguous speech, for which typically a label distribution (or so-called soft labels) is available. To address these issues, we propose in this paper a novel SER model that explicitly handles ambiguous speech samples and overcomes the shortcomings outlined above. Central to our approach is a novel real-time soft-label correction strategy designed to refine the annotations assigned to ambiguous speech. The proposed model leverages both, (explicitly) labeled as well as ambiguous samples and applies the dynamic soft-label correction strategy alongside an enhanced inter-class difference loss function to iteratively optimize the label distributions during training. We theoretically demonstrate that our method is capable of approximating the true emotional distribution of speech even in the presence of label noise, suggesting that utilizing ambiguous speech samples without explicit emotion labels still contributes toward more effective emotion recognition. Furthermore, we integrate the representational power of convolutional neural networks (CNNs) with the contextual modeling capabilities of Wav2Vec 2.0 to enable a comprehensive extraction of spatio-temporal speech features. Experimental results on the IEMOCAP multi-label dataset confirm the effectiveness of our approach, achieving state-of-the-art performance with significant improvements in weighted accuracy (WA) and unweighted accuracy (UA) over competing methods.

Abstract In the Internet of Medical Things (IoMT), the vulnerability of federated learning (FL) to single points of failure, low‐quality nodes, and poisoning attacks necessitates innovative solutions. This article introduces a FL‐driven dual‐blockchain approach to address these challenges and improve data sharing and reputation management. Our approach comprises two blockchains: the Model Quality Blockchain (MQchain) and the Reputation Incentive Blockchain (RIchain). MQchain utilizes an enhanced Proof of Quality (PoQ) consensus algorithm to exclude low‐quality nodes from participating in aggregation, effectively mitigating single points of failure and poisoning attacks by leveraging node reputation and quality thresholds. In parallel, RIchain incorporates a reputation evaluation, incentive mechanism, and index query mechanism, allowing for rapid and comprehensive node evaluation, thus identifying high‐reputation nodes for MQchain. Security analysis confirms the theoretical soundness of the proposed method. Experimental evaluation using real medical datasets, specifically MedMNIST, demonstrates the remarkable resilience of our approach against attacks compared to three alternative methods.

Cyber propaganda has become an increasingly sophisticated tool for manipulating public perception and discourse within online social networks (OSNs). The effectiveness of cyber propaganda is strongly influenced by the interplay between individual awareness and the underlying topology of OSNs that facilitates the spread of propaganda. However, existing interventions primarily focus on continuous control strategies, which may not be feasible in certain real-world scenarios. Therefore, effectively suppressing the spread of cyber propaganda while taking into account the above impact factors remains a challenging problem. In this study, we propose a methodology that combines the optimal impulse control (OIC) theory with a novel propagation model to address this problem. Our propagation model is the first to take into account the effects of the cognitive differences and interconnectivity of OSNs on the dynamics of cyber propaganda. By employing the OIC framework and our newly developed propagation model, we formulate an OIC problem. The goal is to find impulse strategies that optimally balance the cost of intervention against its effectiveness. Using the impulse maximum principle, we establish the necessary conditions for optimal impulse strategies and construct an algorithm to solve the OIC problem. Our numerical experiments, conducted on three distinct social networks, demonstrated that: 1) awareness levels play a crucial role in effectively suppressing the spread of cyber propaganda on OSNs; and 2) our impulse strategies are significantly superior to random strategies in terms of suppression effect, thereby evidencing their cost-effectiveness.

Online social networks (OSNs) facilitate the rapid and extensive spreading of rumors. While most existing methods for debunking rumors consider a solitary debunker, they overlook that rumor-mongering and debunking are interdependent and confrontational behaviors. In reality, a debunker must consider the impact of rumor-mongering behavior when making decisions. Moreover, a single rumor-debunking strategy is ineffective in addressing the complexity of the rumor environment in networks. Therefore, this article proposes a hybrid rumor-debunking approach that combines truth dissemination and regulatory measures based on the differential game theory under adversarial behaviors of rumor-mongering and debunking. Toward this end, we first establish a rumor propagation model using node-based modeling techniques that can be applied to any network structure. Next, we mathematically describe and analyze the processes of rumor-mongering and debunking. Finally, we validate the theoretical results of the proposed method through various comparative experiments, including comparisons with a random strategy, a uniform strategy, and single strategy models on real-world datasets collected from Facebook, Twitter, and YouTube. Furthermore, we harness two actual rumor events to estimate parameters and predict rumor propagation, thereby affirming the veracity and effectiveness of our rumor propagation model.

Existing visual-textual sentiment analysis methods usually get poor performance due to limited utilization of the correlation between different modalities, i.e., they neglect the heterogeneity and homogeneity of different modalities. To overcome these limitations, we propose a Multimodal Fusion Network (called MFN) with a multi-head self-attention mechanism. MFN can minimize noise interference between different modalities through neural networks and attention mechanisms to obtain independent visual and textual features. Furthermore, it can exploit correlations between fine-grained local region feature representations from multimodal with different numbers of hidden neurons to leverage complementary information from heterogeneous visual and textual data. Extensive experiments show MFN outperforms the 11 state-of-the-art methods by at least 0.11%, 0.13%, and 0.38% on Twitter, Flickr, and Getty image datasets, respectively.

Conversational emotion recognition represents an important machine-learning problem with a wide variety of deployment possibilities. The key challenge in this area is how to properly capture the key conversational aspects that facilitate reliable emotion recognition, including utterance semantics, temporal order, informative contextual cues, speaker interactions as well as other relevant factors. In this paper, we present a novel Graph Neural Network approach for conversational emotion recognition at the utterance level. Our method addresses the outlined challenges and represents conversations in the form of graph structures that naturally encode temporal order, speaker dependencies, and even long-distance context. To efficiently capture the semantic content of the conversations, we leverage the zero-shot feature-extraction capabilities of pre-trained large-scale language models and then integrate two key contributions into the graph neural network to ensure competitive recognition results. The first is a novel context filter that establishes meaningful utterance dependencies for the graph construction procedure and removes low-relevance and uninformative utterances from being used as a source of contextual information for the recognition task. The second contribution is a feature-correction procedure that adjusts the information content in the generated feature representations through a gating mechanism to improve their discriminative power and reduce emotion-prediction errors. We conduct extensive experiments on four commonly used conversational datasets, i.e., IEMOCAP, MELD, Dailydialog, and EmoryNLP, to demonstrate the capabilities of the developed graph neural network with context filtering and error-correction capabilities. The results of the experiments point to highly promising performance, especially when compared to state-of-the-art competitors from the literature.

Micro-expression recognition (MER) is a challenging computer vision problem, where the limited amount of available training data and insufficient intensity of the facial expressions are among the main issues adversely affecting the performance of existing recognition models. To address these challenges, this paper explores a transfer–learning enabled MER model using a densely connected feature extraction module with mixed attention. Unlike previous works that utilize transfer learning to facilitate MER and extract local facial-expression information, our model relies on pretraining with three diverse macro-expression datasets and, as a result, can: ( i ) overcome the problem of insufficient sample size and limited training data availability, ( i i ) leverage (related) domain-specific information from multiple datasets with diverse characteristics, and ( i i i ) improve the model adaptability to complex scenes. Furthermore, to enhance the intensity of the micro-expressions and improve the discriminability of the extracted features, the Euler video magnification (EVM) method is adopted in the preprocessing stage and then used jointly with a densely connected feature extraction module and a mixed attention mechanism to derive expressive feature representations for the classification procedure. The proposed feature extraction mechanism not only guarantees the integrity of the extracted features but also efficiently captures local texture cues by aggregating the most salient information from the generated feature maps, which is key for the MER task. The experimental results on multiple datasets demonstrate the robustness and effectiveness of our model compared to the state-of-the-art.

Dialogic emotion analysis is an emerging and important research field in natural language processing. It aims to understand and process emotions in various forms of dialogue, such as human-human conversations, human–machine interactions, and chatbot responses. However, dialogic emotion analysis faces many challenges, such as the diversity of dialogue genres, the complexity of emotional expressions, and the difficulty of capturing the emotional needs of dialogue participants. Moreover, the current dialogue systems lack the ability to analyze emotions effectively and appropriately in different dialogue contexts. Therefore, a comprehensive review of the existing research on dialogic emotion analysis is needed. This survey aims to review dialogic emotion analysis methods based on natural language processing from 2017 to 2024. The review process follows the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). We summarize the research methods and emphasize their main research contributions. In addition, we also discuss current research trends and possible future research directions, as well as the impact of personal traits on emotions and potential ethical issues.

Cyber propaganda wars significantly impact users on Online Social Networks (OSNs), potentially altering their psychological/ideological attitudes and behaviors. Understanding these behavioral dynamics necessitates models that can effectively capture the propagation of dual competitive information, encompassing both propaganda and counter-propaganda campaigns by both conflicting parties. However, current models do not adequately account for competitive information spreading in dual setting and it lacks efficient strategies for managing both propaganda and counter-propaganda investments. To bridge these gaps, our study presents an innovative netwORked dIfferENTial gAme wiTh hybrId cONtrol (ORIENTATION) framework that integrates differential game with 1) a degree-based network model characterizing the spreading dynamics of dual competitive information for both parties; and 2) a dual hybrid control mechanism consisting of investment rates by continuous-time propaganda and discrete-time counter-propaganda. Using this framework, we formulate the Hybrid-contrOlled Differential GamE (HODGE) problem. We theoretically derive the necessary conditions for Nash equilibrium, and develop an iterative algorithm, termed theHODGEalgorithm, to numerically approximate the Nash equilibrium. Our experiments, performed on different groups of OSNs, reveal that the resulting strategy profiles consistently outperform several alternative profiles in terms of cost-effectiveness. Scalability assessment for theHODGEalgorithm is then carried out on OSNs with different scales, demonstrating its strong performance in terms of computational efficiency, scalability and practicability. Additional experimental results suggest that a decrease in the lower bounds of the investment rates in both propaganda and counter-propaganda campaigns and an early implementation of counter-propaganda strategies can significantly enhance cost-effectiveness, offering strategic insights for those engaged in cyber propaganda war.

Despite extensive research on optimal control formulations for cyber threat mitigation, a significant gap persists between theoretical and practical implementation in real-time scenarios. The open-loop structure of the optimal control framework is insufficiently robust for effectively addressing cyber threats. To overcome this, adopting a model learning process that iteratively updates the optimal control strategy is proposed. This paper proposes an innovative approach to addressing cybersecurity attacks in the Internet of Things (IoT) networks by integrating reinforcement learning (RL) and model predictive control (MPC) in a hybrid framework to optimize control parameters and enhance system effectiveness in combating malware. This novel approach aims to overcome the limitations of the previous approaches and establish superior control strategies for IoT network security. This approach enhances the adaptability and responsiveness of the mitigation process, improving the handling of evolving cyber threats in real-world applications. This framework enhances the security and resilience of IoT networks against malicious activities, offering a robust solution for mitigating cyber threats by leveraging RL algorithms and the proactive capabilities of MPC. A comprehensive evaluation demonstrates the effectiveness and efficiency of the hybrid framework, highlighting its potential to protect IoT networks from evolving cybersecurity risks. The primary aim extends beyond using an RL agent solely for computing control actions to optimize closed-loop performance and stability. It also leverages RL to estimate model parameters that are currently unknown but within known bounds. Our main objective in using the RL agent is to accurately estimate unidentified model parameters within specified limits. The simulation results provide compelling evidence supporting the effectiveness of this methodology in mitigating malware propagation, highlighting its superior performance compared to state-of-the-art methods. RLMPC rapidly initiated recovery, achieving full network restoration in 8 seconds and recovering 60 IoT devices. Also, the evaluation focused on average speed, scalability, and performance under various cyber-attack scenarios.

Regarding information dissemination in Device-to-Device (D2D) communications, most existing works only consider user awareness as the influencing factor, which cannot accurately reflect the interaction between users and devices. In fact, user awareness and information interaction are inseparable from the social tie that describes the users’ intimacy. In this paper, we propose a social tie-driven coupling propagation dynamical model, taking into account both user awareness diffusion and information transmission. This model includes interprocess interaction factors that fully describe the user-user and user-device interaction behaviors. Through intensive experiments on the real-world Peer-to-Peer (P2P) and Facebook datasets with a tailored propagation algorithm, we show that the proposed model has a higher propagation speed and covers a larger propagation range than that of an existing single propagation process model. Our work also reveals that a strong social tie between users will promote the transmission of device information, which further accelerates the diffusion of user awareness.

Speech, as a necessary way to express emotions, plays a vital role in human communication. With the continuous deepening of research on emotion recognition in human–computer interaction, speech emotion recognition (SER) has become an essential task to improve the human–computer interaction experience. When performing emotion feature extraction of speech, the method of cutting the speech spectrum will destroy the continuity of speech. Besides, the method of using the cascaded structure without cutting the speech spectrum cannot simultaneously extract speech spectrum information from both temporal and spatial domains. To this end, we propose a spatial–temporal parallel network for speech emotion recognition without cutting the speech spectrum. To further mix the temporal and spatial features, we design a novel fusion method (called multiple fusion) that combines the concatenate fusion and ensemble strategy. Finally, the experimental results on five datasets demonstrate that the proposed method outperforms state-of-the-art methods.

In vehicular ad hoc networks (VANETs), there are many resource constrained communication nodes, distributed geographical locations, and low delay requirements. The authentication system is required to be distributed. However, the information interaction between traditional distributed authentication entities (AEs) is not transparent, and the use of blockchain is only to save certificate status records on the chain. There is a lack of trusted security management mechanisms between AEs, so there is a risk of collusion attacks and behavior record forgery and tamper attacks between AEs. Therefore, this paper mainly addresses the management problems of opaque interaction and insufficient security verification of distributed authentication entities in the VANETs scenario. In this paper, based on the security features of blockchain technology, such as public audit and tamper proof block structure, and distributed trust consensus, we propose a hierarchical and trust certificate service chain based on reputation value called HCSC. We have a set of safe and reliable management mechanisms between distributed authentication entities such as the Bitcoin system. First, we integrate the traditional hierarchical authentication system and the blockchain network into a new hierarchical certificate service chain. Then, we propose a reputation evaluation model based on the logistic regression model by quantifying the behavior records of the AEs. In addition, a new hierarchical reputation consensus based on delegated proof of stake (DPoS) and proof of work (PoW) is given. Then, a fast verification based on block height and security verification for the efficiency and security of certificate verification is proposed. Finally, the security analysis is given, and a prototype implementation of HCSC is developed based on Fabric. The experimental simulation and security analysis reveal that the HCSC is very efficient and suitable for the above issues.

Speech emotion recognition has always been a challenging task due to the difference in emotion expression and perception. Currently, in the supervised speech emotion recognition systems, the soft label overcomes the disadvantage of the hard label losing annotations variability and emotion perception subjectivity, but it only considers the emotion perceptions of a few annotators and thus still brings high statistical error. For this issue, this paper redefines the target and designs a novel loss function (denoted as inter-class difference loss), which enables the network to adaptively learn an emotion distribution in all utterances. This not only restricts the negative class probability less than the positive class probability, but also limits the negative class probability close to zero. To make the speech emotion recognition system more efficient, this paper proposes an end-to-end network, called response residual network (R-ResNet), which incorporates the ResNet for features extraction, together with the emotion response module for data augmentation and variable-length data processing. Finally, the experimental results not only demonstrate the advanced performance of our work, but also confirm that the ambiguous utterances contain emotional characteristics. In addition, another interesting finding is that, on the unbalanced dataset, the batch normalization (BN) after addition performs better than BN before addition.

To balance the trade-off between contextual information and fine-grained information in identifying specific emotions during a dialogue and combine the interaction of hierarchical feature related information, this paper proposes a hierarchical feature interactive fusion network (named DHF-Net), which not only can retain the integrity of the context sequence information but also can extract more fine-grained information. To obtain a deep semantic information, DHF-Net processes the task of recognizing dialogue emotion and dialogue act/intent separately, and then learns the cross-impact of two tasks through collaborative attention. Also, a bidirectional gate recurrent unit (Bi-GRU) connected hybrid convolutional neural network (CNN) group method is designed, by which the sequence information is smoothly sent to the multi-level local information layers for feature exaction. Experimental results show that, on two open session datasets, the performance of DHF-Net is improved by 1.8% and 1.2%, respectively.

Considering the horizontal and vertical propagation of computer viruses over the Internet, this article proposes a hybrid susceptible-latent-breaking-recovered-susceptible (SLBRS) model. Through mathematical analysis of the model, two equilibria (virus-free and virose equilibria) and their global stabilities are both proved depending on the basic reproduction number R 0 , which is affected by the vertical propagation of infected computers. Moreover, the feasibility of the obtained results is verified by numerical simulations. Finally, the dependence of R 0 on system parameters and the parameters affecting the stability level of infected computers are both analyzed.

Abstract In massive multiple‐input multiple‐output (MIMO) systems, most of the existing detection work mainly assumes that the channel is the additive white Gaussian noise (AWGN). However, this assumption is difficult to apply to practical communication scenarios. To this end, this paper proposes a message passing detection (MPD) algorithm with a convolutional neural network (CNN) (denoted as iterative MPD‐CNN structure) under correlated noise channels, which is helpful to solve the issue of detection performance degradation in non‐ideal AWGN channels. Firstly, the MPD algorithm based on the channel hardening phenomenon is used to initially estimate the transmitted signal, and then the CNN is concatenated to remove the estimation error for obtaining more accurate channel noise, which provides a beneficial noise distribution for the MPD algorithm. Finally, the theoretical analysis and simulation results show that the proposed iterative MPD‐CNN structure can improve the detection performance in conditions of correlated noise channels and fewer antennas. Compared with the traditional MPD algorithm, its detection performance is more superior.

Human activity recognition plays a fundamental role in smart home systems and contributes to remote health monitoring for the elderly or disabled. However, the shallow architecture and heavy parameters of the current 3-D convolutional networks (3-D ConvNets) still restrict the recognition efficiency and spatiotemporal representations. For these issues, this article proposes a framework called LMFNet, which is mainly composed of a deep attentive 3-D residual network (A3D ResNet) and a multistage fusion strategy. Specifically, LMFNet changes the information flow of transmission in the C3D network and implements a residual learning method for efficient training. Besides, the two-stream-fused spatiotemporal attention 3-D ConvNets (2S-FSTA3DCN) are built based on the A3D ResNet. The experimental results show that the proposed LMFNet can achieve a higher recognition accuracy and satisfactory training efficiency compared with the existing methods.

In massive multiple‐input multiple‐output (MIMO) systems, the channel state information (CSI) feedback enables performance gain in frequency division duplex networks. However, with the increase in the number of antennas, the feedback overhead of CSI will also enhance. To this end, this study addresses the issue of massive MIMO CSI reconstruction using convolutional neural network (CNN), long short‐term memory (LSTM) and attention mechanism, and proposes an efficient network architecture (denoted as CNN‐LSTM‐A). To achieve a compromise between performance and complexity, the proposed method significantly reduces the number of training parameters by utilising a single‐stage network rather than a multiple‐stage network. Finally, simulation results show that the authors method can reduce the feedback overhead of CSI effectively, and achieves better performance in terms of CSI compression and recovery accuracy compared with existing state‐of‐the‐art methods.