Recent advances have emphasized the importance of meme stickers in open-domain dialogue systems.However, previous studies overlook the one-to-many issue that a single sticker could represent various emotions in different dialogue contexts.Additionally, they require retraining the model for new stickers which did not appear in previous training.To address the above issues, we propose in this paper an Emotion-Aware and Efficient Meme Sticker Dialogue generation framework.In the framework, we design an Emotion Adaptive Prompt to capture the emotional cues from the dialogue history, which is sent to an Emotion-Aware Fusion Decoder to guide the generation of text responses and to a meme sticker selector to choose the corresponding sticker.Furthermore, to improve the stickers' selection efficiency, we further incorporate the few-shot learning strategy into the proposed framework to avoid extensive model retraining for unseen meme stickers.Through extensive experiments, we demonstrate the superior performance of the proposed E 2 MSD compared to existing methods regarding the quality of response generation and the efficiency of meme sticker retrieval.

Multimodal Dialogue agents are often required to respond to conversation history using both textual and visual content. Even though current dialogue studies predominantly strive to generate natural texts or images, they fall short in considering the relevance of multimodal responses within a dialogue context, consequently confining agents from making prudent choices based on multiple alternatives and their associated relevance scores for decision-making. In this paper, we present a bidirectional multimodal dialogue framework that skillfully combines the forward generation of multiple text and image response candidates with reverse selection guided by relevance scores evaluated on dialogue context, facilitating agents in selecting the most suitable multimodal responses. Specifically, the forward generation aspect of our framework leverages a stage-wise approach, first producing textual replies and composite visual descriptions from the dialogue context, followed by the generation of visual responses aligned with the descriptions. In the reverse selection process, visual responses are translated into tangible descriptive texts that, in conjunction with textual responses, are inversely tied back to the dialogue context for relevance assessment, assigning a reference score to each multimodal response candidate to assist the intelligent agent in making informed decisions. Experimental outcomes demonstrate that our proposed bidirectional dialogue response framework markedly elevates performance in both automatic and human evaluations, yielding a range of contextually fitting multimodal responses for selection.

The easy sharing of multimedia content on social media has caused a rapid dissemination of fake news, which threatens society’s stability and security. Therefore, fake news detection has garnered extensive research interest in the field of social forensics. Current methods primarily concentrate on the integration of textual and visual features but fail to effectively exploit multi-modal information at both fine-grained and coarse-grained levels. Furthermore, they suffer from an ambiguity problem due to a lack of correlation between modalities or a contradiction between the decisions made by each modality. To overcome these challenges, we present a Multi-grained Multi-modal Fusion Network (MMFN) for fake news detection. Inspired by the multi-grained process of human assessment of news authenticity, we respectively employ two Transformer-based pre-trained models to encode token-level features from text and images. The multi-modal module fuses fine-grained features, taking into account coarse-grained features encoded by the CLIP encoder. To address the ambiguity problem, we design uni-modal branches with similarity-based weighting to adaptively adjust the use of multi-modal features. Experimental results demonstrate that the proposed framework outperforms state-of-the-art methods on three prevalent datasets.

Symmetric nonnegative matrix factorization (SNMF) is effective to cluster nonlinearly separable data, which uses the constructed graph to capture the structure of inherent clusters. Nevertheless, many SNMF-based clustering approaches implicitly enforce either the sparseness constraint or the smoothness constraint with the limited supervised information in the form of cannot-link or must-link in a semi-supervised manner, which may not be quite satisfactory in many applications where sparseness and smoothness are demanded explicitly and simultaneously. In this paper, we propose a new semi-supervised SNMF-based approach termed Semi-supervised Structured SNMF-based clustering (S3NMF). The method flexibly enforces the block-diagonal structure to the similarity matrix, where the sparseness and smoothness are simultaneously considered, so that we can obtain the desirable assignment matrix by simultaneously learning similarity and assignment matrices in a constrained optimization problem. We formulate S3NMF with a semi-supervised manner and utilize the indirect constraints of sparseness and smoothness by cannot-link and must-link. To effectively solve S3NMF, we present an alternating iterative algorithm with theoretically proved convergence to seek for the solution of the optimization problem. Experiments on five benchmark data sets show better performance and satisfactory stability of the proposed method.