Multi-Modal Sarcasm Detection (MSD) aims to combine multiple modal information to identify implicit sarcastic sentiment. However, the significance of commonsense knowledge in implicit emotion recognition has been frequently overlooked. Additionally, the important visual emotions associated with textual modal sarcastic cues are typically dispersed throughout the entire image, making it difficult to accurately focus on crucial sarcastic features. Therefore, we propose a Knowledge-Aware Focused Graph Convolutional Networks (KFGC-Net) to tackle these issues. Specifically, we first construct a cross-modal knowledge-aware graph based on commonsense concepts for each instance. This graph explicitly establishes connections between significant visual sentiments and relevant textual tokens of knowledge. Next, we integrate the transformer encoder optimized with convolutional operations with the Convolutional Block Attention Module to compensate for the model’s lack of attention to important features. Finally, we design a Global Modality Synergistic Fusion (GMSF) block, aiming to model the global relationships in each modality for complementing global sarcasm detection result. Notably, we analyze the proposed framework by testing it on several benchmark datasets, and the results outperform the existing state of the art.

Understanding emotions in dialogue is an essential part of human communication, its an extremely complex cognitive process involving cross-modal interactions and cross-emotional associations, and multimodal sarcasm detection is an emerging but challenging research task in this process aiming at video discourse incorporating appropriate contextual information and external knowledge and identifying sarcasm by understanding both verbal and non-verbal components. However, existing research primarily focuses on constructing multimodal fusion representations and capturing incongruity between modalities as indicative cues for recognizing sarcasm, which relies on a fixed network design architecture that is difficult to cope with complex and diverse satirical scenarios in real life. As humans, we rely on the combination of visual and auditory cues, such as facial expressions and intonations, to understand information. Our brains are implicitly trained to integrate information from multiple senses to form a comprehensive understanding of conveyed messages, a process known as multi-sensory integration. The combination of different modalities not only provides additional information but also amplifies the information conveyed by each modality relative to others. Therefore, dynamic variations in the weights of different modalities play a crucial role in multi-modal understanding. From this perspective, we propose a new framework called Multi-view BART(MV-BART), which is capable of exploiting multi-granularity cues from multiple viewpoints and dynamically adjusting the view weights, applied to different sarcastic scenarios. It is worth mentioning that we analyze the proposed framework by testing it on several benchmark datasets, and the results outperform the existing state-of-the-art.

Multi-modal Sarcasm Detection (MSD) combines multiple modalities to identify implicit sarcastic sentiment, but commonsense knowledge's role in emotion recognition is often overlooked. Visual emotions tied to sarcastic cues in text are usually dispersed across the image, complicating detection. we propose a Dual Synergetic Perception Graph Convolutional Networks (DSP-GCN) to address these issues. First, we create a cross-modal knowledge incongruity graph linking key visual sentiments and relevant text tokens. Then, we enhance feature focus using a transformer encoder with the Convolutional Block Attention Module. Finally, the Global Modality Synergistic Fusion (GMSF) block models global relationships in each modality for improved sarcasm detection. Note that, our framework outperforms state-of-the-art methods on benchmark datasets.