In this work, we tackle the multi-modal irony detection problem and propose a lightweight approach that combines pretrained multi-modal and text-based smaller language models. The proposed approach makes use of generated image captions and original text content to fine-tune a transformer based model for text classification. In the proposed solution, Vision-Language Model (VLM) is used for generating image captions, without any fine-tuning. On two benchmark datasets, the experiments demonstrate the benefit of the proposed method against the baseline solutions.

The rapid dissemination of misinformation through social media increased the importance of automated fact-checking. Furthermore, studies on what deep neural models pay attention to when making predictions have increased in recent years. While significant progress has been made in this field, it has not yet reached a level of reasoning comparable to human reasoning. To address these gaps, we propose a multi-task explainable neural model for misinformation detection. More specifically, this work formulates an explanation generation process of the model’s veracity prediction as a text summarization problem. Additionally, the performance of the proposed model is discussed on publicly available datasets and the findings are evaluated with related studies.