Memes possess a humorous intent, yet they can also be used for malicious purposes. Analysing meme data has the potential to enhance content monitoring, identify emerging topics, and support content moderation in online platforms. Memes also represent an interesting use case for multimodal machine learning, as they combine text and image data. In this study, we explored the linguistic characteristics and analysed the convergent themes of five meme classes through common word extraction. Moreover, we compared the effectiveness of various machine learning models, i.e., unimodal (text or image) and multimodal (early fusion, late fusion) in binary and multiclass meme classification tasks. Our results on a large meme dataset showed that memes heavily adhered to current affairs, demonstrated by the high frequency of topical words across meme classes. Regarding model accuracy, early fusion achieved superior accuracy over late fusion in meme classification. Binary models outperformed multi-class classification methods. However, fusion models did not consistently surpass the accuracy of independent text or image-based models.

Social media content is increasingly subject to hate speech towards specific demographic groups. In this context, machine learning approaches appear relevant to detect malicious contents and support moderators in mitigation initiatives. However, many of the existing approaches are exclusively focused on the analysis of textual contents. On the other hand, studies that address multiple data modalities often rely on a single feature representation and a fully supervised learning setting. In this paper, we tackle multimodal hate speech detection resorting to different learning settings (one-class learning and binary classification). We also investigate the effectiveness of multiple deep learning model backbones and language models to extract embedding feature representations for text and image modalities. Our experiments with a real-world hate speech dataset show that there is a significant performance gap between one-class learning and binary classification, and that the choice of embedding representations for image and text modalities can impact the detection performance for different predictive models.

Social media platforms are pivotal in information dissemination but also contribute to the rapid spread of misinformation, especially during high-impact events like natural disasters, terrorist attacks, and political unrest. While recent advances in multi-modal learning have enhanced misinformation detection by integrating features from various modalities (e.g., text, images), certain areas remain under-explored, particularly the use of event-based multi-modal data. This paper introduces a novel approach to misinformation detection on social media using an event-based multi-modal learning framework. Our method extends beyond traditional techniques by employing latent variable modeling to capture non-linear associations in event-based multi-modal data and to generate joint features between events for classification. This approach enhances misinformation detection and enables the contextual understanding of terms across different events. We provide a detailed analysis of our dataset preparation, methodology, and results, demonstrating the effectiveness of our framework on a widely-used dataset of tweets from high-impact events. The paper concludes with insights into potential enhancements and future directions in multi-modal misinformation detection.

Social media content can present a number of threats, including misinformation and hate speech towards specific demographic groups. One challenge is to effectively discriminate between benign and malicious posts, given the massive amount of available content. In this context, predictive models for malicious content detection can be extremely valuable, leading to the automatic removal of posts and user accounts or content being flagged for subsequent moderation. However, some of the existing detection models are limited to the analysis of a single data modality. At the same time, most multi-modal approaches operate in a fully supervised learning setting that assumes the availability of labeled data for both benign and malicious content. In this paper, we fill this gap by proposing a multimodal one-class learning approach for malicious online content detection. Our approach leverages feature extraction, dimensionality reduction, and one-class learning models to analyze text and image data in online posts simultaneously. Models learn their decision function in the challenging scenario where only benign online content is used as training data, overcoming the limitations of a fully supervised setting. Our experiments with two real-world datasets containing misinformation and hate speech posts reveal the effectiveness of different combinations of one-class learning models and dimensionality reduction techniques.