Multi-modal sarcasm detection aims to identify whether a given image-text pair is sarcastic. The pivotal factor of the task lies in accurately capturing incongruities from different modalities. Although existing studies have achieved impressive success, they primarily committed to fusing the textual and visual information to establish cross-modal correlations, overlooking the significance of original unimodal incongruity information at the text-level and image-level. Furthermore, the utilized fusion strategies of cross-modal information neglected the effect of inherent ambiguity within text and image modalities on multimodal fusion. To overcome these limitations, we propose a novel Ambiguity-aware Multi-level Incongruity Fusion Network (AMIF) for multi-modal sarcasm detection. Our method involves a multi-level incongruity learning module to capture the incongruity information simultaneously at the text-level, image-level and cross-modal-level. Additionally, an ambiguity-based fusion module is developed to dynamically learn reasonable weights and interpretably aggregate incongruity features from different levels. Comprehensive experiments conducted on a publicly available dataset demonstrate the superiority of our proposed model over state-of-the-art methods.

Multimodal sentiment detection aims to classify the sentiment polarity of a given imagetext pair.Existing approaches apply the same fixed framework to all input samples, lacking the flexibility to adapt to different image-text pairs.Furthermore, the interaction patterns of these methods are overly homogenized, limiting the model's capacity to extract multimodal sentiment information effectively.In this paper, we develop a Dual-Branch Dynamic Routing Network (D 2 R), which is the first multimodal dynamic interaction model towards multimodal sentiment detection.Specifically, we design six independent units to simulate inter-and intramodal information interactions without depending on any existing fixed frameworks.Additionally, we configure a soft router in each unit to guide path generation and introduce the path regularization term to optimize these inference paths.Comprehensive experiments on three publicly available datasets demonstrate the superiority of our proposed model over state-ofthe-art methods.

Multimodal named entity recognition (MNER) aims to detect named entities and identify the entity types based on texts and attached images, which also generates inputs for other comprehensive tasks, such as multimodal machine translation, visual dialog, and multimodal sentiment analysis. Existing studies have limitations in text-image matching and multimodal semantic disparity reduction. For one thing, current methods fail to resolve both overall and local text-image matching issues in a self-guided way. For another, the static graphs constructed in MNER models are challenging in bridging the semantic gap between different modalities. In this work, a dynamic graph construction framework (DGCF) is proposed to solve the above-mentioned limitations. A similarity vector-based text-image matching inferring strategy is designed to obtain the overall and local matching relation between text and image while the overall matching determines the retained proportion of visual information. Then, a multimodal dynamic graph interaction module is developed. Within each layer of the module, the local matching relations and part of speech (POS)-based multihead attention are integrated to construct a dynamic cross-modal graph and a semantic graph. Lastly, a CRF layer is used to predict entity label. Extensive experiments are performed on two benchmark datasets. The experimental results reveal that our model is a competitive alternative and achieves state-of-the-art performance.

Multimodal Named Entity Recognition (MNER) on social media posts is a leading but challenging task. However, most existing MNER methods fail to effectively exploit the visual information from the image. Besides, the multimodal interaction and alignment remains unsettled. In this paper, we propose a novel token-wise graph-based framework to deal with the MNER task. Specifically, a token-wise image processing manner is established. A muti-modal graph is constructed based on the textual token derived from BERT and the visual token derived from SwinT. Then, the muti-modal graph is fed into a multi-layer Transformer-based module for intra- and inter-modal information fusion. In addition, multiple contrastive learning is devised to perform the global and local alignment between textual and visual nodes. Experimental results on two benchmark multimodal datasets indicate that our model achieves state-of-the-art performance in MNER tasks.