Recent progress in aligning images with texts has achieved remarkable results, however, existing models tend to serve general queries and often fall short when dealing with detailed query requirements. In this paper, we work towards Entity-centric Image-Text Matching (EITM), a finer-grained image-text matching task that aligns texts and images centered around specific entities. The main challenge in EITM lies in bridging the substantial semantic gap between entity-related information in texts and images, which is more pronounced than in general image-text matching problems. To address this challenge, we adopt CLIP as our foundational model and devise a Multimodal Attentive Experts (MMAE)-based contrastive learning to adapt CLIP into an expert for EITM problem. Particularly, the core of our multimodal attentive experts learning is to generate explanation texts by Large Language Models (LLMs) as bridging clues. In specific, we first employ off-the-shelf LLMs to generate explanatory text. This text, along with the original image and text, is then fed into our Multimodal Attentive Experts module to narrow the semantic gap within a unified semantic space. Upon the enriched feature representations generated by MMAE, we have further developed an effective Gated Integrative Image-text Matching (GI-ITM) strategy. GI-ITM utilizes an adaptive gating mechanism to combine features from MMAE, followed by applying image-text matching constraints to enhance the alignment precision. Our method has been extensively evaluated on three social media news benchmarks: N24News, VisualNews, and GoodNews. The experimental results demonstrate that our approach significantly outperforms competing methods. Our code is available at: https://github.com/wangyxxjtu/ETE.

Fact verification task has emerged as an essential research topic recently due to abundant fake news spreading on the Internet. The task based on unstructured data (i.e., news) has achieved great development, but the task based on structured data (i.e., table) is still in the primary development period. The existing methods usually construct complete heterogeneous graph networks around statement, table, and program subgraphs, and then infer to learn similar semantics on them for fact verification. However, they generally connect the nodes with the same content between subgraphs directly to frame a larger graph network, which has serious sparsity in connections, especially when subgraphs possess limited semantics. To this end, we propose tight-fitting graph inference network (TFGIN), which innovatively builds tight-fitting graphs (TF-graphs) to strengthen the connections of subgraphs and designs inference modeling layer (IML) to learn coherence evidence for fact verification. Specifically, different from traditional connection ways, the constructed TF-graph enhances inter-graph and intra-graph connections of subgraphs through subgraph segmentation and interaction guidance mechanisms. IML could reason the semantics with strong correlation and high consistency as explainable evidence. Experiments on three competitive datasets confirm the superiority and scalability of our TFGIN.

The existing models for multi-modal fake news detection focus mainly on capturing common similar semantics between different modalities to improve detection performance. However, they ignore the extraction of inconsistent features between these modalities. The intuitive cognition way people identify a piece of fake news is generally to discover if there are inconsistent semantics among news content itself and its comments, which could be abstracted as “comparing news image-text consistency - finding valuable comments - reasoning in-/consistency between news and comments”. Inspired by the cognitive process, we propose Human Cognition-based Consistency Inference Networks (HCCIN) to comprehensively explore consistent and inconsistent semantics for multi-modal fake news detection. Specifically, we first design cross-modal alignment layer to learn consistent semantics between textual and visual information within the multi-modal news, and then the comment clue discovery layer is devoted to ascertaining the most-concerned semantics by audiences between comments. Finally, we develop collaborative inference layer to drive news consistent semantics and the most-concerned semantics to reason and discover consistent and inconsistent information between them. Experiments on three public datasets, including Weibo, Twitter, and PHEME, reveal the superiority of our HCCIN.

The existing data-driven approaches typically capture credibility-indicative representations from relevant articles for fake news detection, such as skeptical and conflicting opinions. However, these methods still have several drawbacks: 1) Due to the difficulty of collecting fake news, the capacity of the existing datasets is relatively small; and 2) there is considerable unverified news that lacks conflicting voices in relevant articles, which makes it difficult for the existing methods to identify their credibility. Especially, the differences between true and fake news are not limited to whether there are conflict features in their relevant articles, but also include more extensive hidden differences at the linguistic level, such as the perspectives of emotional expression (like extreme emotion in fake news), writing style (like the shocking title in clickbait), etc., the existing methods are difficult to fully capture these differences. To capture more general and wide-ranging differences between true and fake news, in this paper, directly from the different categories of news itself, we propose a Category-controlled Encoder-Decoder model (CED) to generate examples with category-differentiated features and extend the dataset capacity to achieve data enhancement effect, thus enhancing fake news detection. Specifically, to make the generated examples enrich more news features, we develop news-guided encoder to guide relevant articles to generate news-semantic context representations. To drive the generated examples to contain more category-differentiated features, we devise category-controlled decoder which relies on pattern-shared unit to respectively capture intra-category shared features within true or fake news, and employs restriction unit to force the two types of shared features to be more different for highlighting inter-category differentiated features. The experimental results on three datasets demonstrate the superiority of CED.