With more and more messages in the form of text and image being spread on the Internet, multi-modal rumor detection has become the focus of recent research. However, most of the existing methods simply concatenate or fuse image features with text features, which can not fully explore the interaction between modalities. Meanwhile, they ignore the convergence inconsistency problem between strong and weak modalities, that is, the dominant rumor text modality may inhibit the optimization of image modality. In this paper, we investigate multi-modal rumor detection from a novel perspective, and propose a Multi-modal Graph Interactive Network with Adaptive Gradient (MGIN-AG) to solve the problem of insufficient information mining within and between modalities, and alleviate the optimization imbalance. Specifically, we first construct fine-grained graph for each rumor text or image to explicitly capture the relation between text tokens or image patches in uni-modal. Then, the cross modal interaction graph between text and image is designed to implicitly mine the text-image interaction, especially focusing on the consistency and mutual enhancement between image patches and text tokens. Furthermore, we extract the embedded text in images as an important supplement to improve the performance of the model. Finally, a strategy of dynamically adjusting the model gradient is introduced to alleviate the under optimization problem of weak modalities in the multi-modal rumor detection task. Extensive experiments demonstrate the superiority of our model in comparison with the state-of-the-art baselines.

Rumors spread through the Internet, especially on Twitter, have harmed social stability and residents’ daily lives. Recently, in addition to utilizing the text features of posts for rumor detection, the structural information of rumor propagation trees has also been valued. Most rumors with salient features can be quickly locked by graph models dominated by cross entropy loss. However, these conventional models may lead to poor generalization, and lack robustness in the face of noise and adversarial rumors, or even the conversational structures that is deliberately perturbed (e.g., adding or deleting some comments). In this paper, we propose a novel Graph Adversarial Contrastive Learning (GACL) method to fight these complex cases, where the contrastive learning is introduced as part of the loss function for explicitly perceiving differences between conversational threads of the same class and different classes. At the same time, an Adversarial Feature Transformation (AFT) module is designed to produce conflicting samples for pressurizing model to mine event-invariant features. These adversarial samples are also used as hard negative samples in contrastive learning to make the model more robust and effective. Experimental results on three public benchmark datasets prove that our GACL method achieves better results than other state-of-the-art models.