Hate speech detection on social media faces challenges in both accuracy and explainability, especially for underexplored Indic languages. We propose a novel explainability-guided training framework, X-MuTeST (eXplainable Multilingual haTe Speech deTection), for hate speech detection that combines high-level semantic reasoning from large language models (LLMs) with traditional attention-enhancing techniques. We extend this research to Hindi and Telugu alongside English by providing benchmark human-annotated rationales for each word to justify the assigned class label. The X-MuTeST explainability method computes the difference between the prediction probabilities of the original text and those of unigrams, bigrams, and trigrams. Final explanations are computed as the union between LLM explanations and X-MuTeST explanations. We show that leveraging human rationales during training enhances both classification performance and the model’s explainability. Moreover, combining human rationales with our explainability method to refine the model’s attention yields further improvements. We evaluate explainability using Plausibility metrics such as Token-F1 and IOU-F1, and Faithfulness metrics such as Comprehensiveness and Sufficiency. By focusing on under-resourced languages, our work advances hate speech detection across diverse linguistic contexts. Our dataset includes token-level rationale annotations for 6,004 Hindi, 4,492 Telugu, and 6,334 English samples.

Detecting sarcasm effectively requires a nuanced understanding of context, including vocal tones and facial expressions. The progression towards multimodal computational methods in sarcasm detection, however, faces challenges due to the scarcity of data. To address this, we present AMuSeD (Attentive deep neural network for MUltimodal Sarcasm dEtection incorporating bi-modal Data augmentation). This approach utilizes the Multimodal Sarcasm Detection Dataset (MUStARD) and introduces a two-phase bimodal data augmentation strategy. The first phase involves generating varied text samples through Back-Translation from several secondary languages. The second phase involves the refinement of a FastSpeech2-based speech synthesis system, tailored specifically for sarcasm to retain sarcastic intonations. Alongside a cloud-based Text-to-Speech (TTS) service, this Fine-tuned FastSpeech2 system produces corresponding audio for the text augmentations. We also evaluate various attention mechanisms for selectively enhancing sarcasm-relevant features, finding self-attention to be the most efficient. Our experiments reveal that the proposed approach achieves a significant F1-score of 81.0% in text-audio modalities, surpassing even models that use three modalities from the MUStARD dataset.

Mohammad Zia Ur Rehman, Anukriti Bhatnagar, Omkar Kabde, Shubhi Bansal, Dr. Nagendra Kumar. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Sarcasm detection poses unique challenges due to the complex nature of sarcastic expressions often embedded across multiple modalities. Current methods frequently fall short in capturing the incongruent emotional cues that are essential for identifying sarcasm in multimodal contexts. In this paper, we present a novel method to capture the pair-wise emotional incongruities between modalities through a cross-modal Contrastive Attention Mechanism (CAM), leveraging advanced data augmentation techniques to enhance data diversity and Supervised Contrastive Learning (SCL) to obtain discriminative embeddings. Additionally, we employ Graph Attention Networks (GATs) to construct modality-specific graphs, capturing intra-modal dependencies. Experiments conducted on the MUStARD++ dataset demonstrate the efficacy of our approach, achieving a macro F1 score of 74.96%, which outperforms state-of-the-art methods.

Dark humor in online memes poses unique challenges due to its reliance on implicit, sensitive, and culturally contextual cues. To address the lack of resources and methods for detecting dark humor in multimodal content, we introduce a novel dataset of 4,379 Reddit memes annotated for dark humor, target category (gender, mental health, violence, race, disability, and other), and a three-level intensity rating (mild, moderate, severe). Building on this resource, we propose a reasoning-augmented framework that first generates structured explanations for each meme using a Large Vision-Language Model (VLM). Through a Role-Reversal Self-Loop, VLM adopts the author's perspective to iteratively refine its explanations, ensuring completeness and alignment. We then extract textual features from both the OCR transcript and the self-refined reasoning via a text encoder, while visual features are obtained using a vision transformer. A Tri-stream Cross-Reasoning Network (TCRNet) fuses these three streams, text, image, and reasoning, via pairwise attention mechanisms, producing a unified representation for classification. Experimental results demonstrate that our approach outperforms strong baselines across three tasks: dark humor detection, target identification, and intensity prediction. The dataset, annotations, and code are released to facilitate further research in multimodal humor understanding and content moderation. Code and Dataset Access: https://github.com/Sai-Kartheek-Reddy/D-Humor-Dark-Humor-Understanding-via-Multimodal-Open-ended-Reasoning

During disasters, access to timely and accurate information is crucial for effective response and recovery efforts. Hashtags have emerged as a lifeline in disaster response, organizing and disseminating critical information on social media, facilitating effective communication and real-time situational awareness. However, existing methods for hashtag recommendation fall short during disasters. Retrieval-based methods rely on fixed predefined hashtag lists, failing to capture dynamic information flow, while generation-based methods lack guidance for generating relevant hashtags. In view of the above, we propose a novel three-stage framework. First, the retriever identifies potential candidate hashtags from a vast collection of tweets annotated with hashtags. Next, a selector narrows down these candidates by analyzing the input tweet and ensuring only the most relevant hashtags are retained. Finally, a diffusion-based seq2seq encoder-decoder generates informative hashtags by leveraging the refined set of candidate hashtags and the original input tweet. The framework infused with diffusion overcomes the limitations of extant encoder-decoder models that produce generic hashtags due to reliance on maximizing training data likelihood. Our diffusion-based approach excels at capturing the dynamic and informal language of disaster situations by reversing a gradual noising process, allowing it to explore wider possibilities and generate more diverse hashtags. We enhance the generator with self-conditioning for better utilization of predicted sequence information. Furthermore, we devise an adaptive nonlinear noise schedule for balanced denoising across time steps for each token in the generated hashtag sequence. Empirical evaluations reveal that our proposed method exhibits superior performance compared to state-of-the-art hashtag recommendation methods in both the quality of generated hashtags and training time.

Social media has emerged as a critical platform for disseminating real-time information during disasters. However, extracting actionable resource data, such as needs and availability, from this vast and unstructured content remains a significant challenge, leading to delays in identifying and allocating resources, with severe consequences for affected populations. This study addresses this challenge by investigating the potential of label and topic features, combined with text embeddings, to enhance the performance and efficiency of resource identification from social media data . We propose Crisis Resource Finder (CRFinder), a novel framework that leverages label encoding and topic features to extract richer contextual information, uncover hidden patterns, and reveal the true context of disaster resources. CRFinder incorporates novel techniques such as multi-level text-label attention and contrastive text-topic attention to capture semantic and thematic nuances within the textual data. Additionally, our approach employs topic injection and selective contextualization techniques to enhance thematic relevance and focus on critical information, which is pivotal for targeted relief efforts. Extensive experiments demonstrate the significant improvements achieved by CRFinder over existing state-of-the-art methods, with average weighted F1-score gains of 7.12%, 6.44%, and 7.89% on datasets from the Nepal earthquake, Italy earthquake, and Chennai floods, respectively. By providing timely and accurate insights into resource needs and availabilities, CRFinder has the potential to revolutionize disaster response efforts.

The recent coronavirus disease (COVID-19) has become a pandemic and has affected the entire globe. During the pandemic, we have observed a spike in cases related to mental health, such as anxiety, stress, and depression. Depression significantly influences most diseases worldwide, making it difficult to detect mental health conditions in people due to unawareness and unwillingness to consult a doctor. However, nowadays, people extensively use online social media platforms to express their emotions and thoughts. Hence, social media platforms are now becoming a large data source that can be utilized for detecting depression and mental illness. However, the existing approaches often overlook data sparsity in tweets and the multimodal aspects of social media. In this article, we propose a novel multimodal framework that combines textual, user-specific, and image analysis to detect depression among social media users. To provide enough context about the user’s emotional state, we propose the following: 1) an extrinsic feature by harnessing the URLs present in tweets and 2) extracting textual content present in images posted in tweets. We also extract five sets of features belonging to different modalities to describe a user. In addition, we introduce a deep learning model, the visual neural network (VNN), to generate embeddings of user-posted images, which are used to create the visual feature vector for prediction. We contribute a curated COVID-19 dataset of depressed and nondepressed users for research purposes and demonstrate the effectiveness of our model in detecting depression during the COVID-19 outbreak. Our model outperforms the existing state-of-the-art methods over a benchmark dataset by 2%–8% and produces promising results on the COVID-19 dataset. Our analysis highlights the impact of each modality and provides valuable insights into users’ mental and emotional states.

With the advent of the Internet, social media platforms have witnessed an enormous increase in user-generated textual and visual content. Microblogs on platforms such as Twitter are extremely useful for comprehending how individuals feel about a specific issue through their posted texts, images, and videos. Owing to the plethora of content generated, it is necessary to derive an insight of its emotional and sentimental inclination. Individuals express themselves in a variety of languages and, lately, the number of people preferring native languages has been consistently increasing. Marathi language is predominantly spoken in the Indian state of Maharashtra. However, sentiment analysis in Marathi has rarely been addressed. In light of the above, we propose an emotion-aware multimodal Marathi sentiment analysis method (MahaEmoSen). Unlike the existing studies, we leverage emotions embedded in tweets besides assimilating the content-based information from the textual and visual modalities of social media posts to perform a sentiment classification. We mitigate the problem of small training sets by implementing data augmentation techniques. A word-level attention mechanism is applied on the textual modality for contextual inference and filtering out noisy words from tweets. Experimental outcomes on real-world social media datasets demonstrate that our proposed method outperforms the existing methods for Marathi sentiment analysis in resource-constrained circumstances.

Users share information on social media platforms by posting visual and textual contents. Due to the massive influx of user-generated content, hashtags are extensively used to manage, organize, and categorize the content. Despite the usability of hashtags, many social media users refrain from assigning hashtags to their posts owing to the uncertainty in choosing appropriate hashtags. Several methods have been proposed to recommend hashtags using content-based information. However, multimodality and personalization aspects of hashtag recommendation have rarely been addressed. In light of the above, we propose a multimoDal pErSonalIzed hashtaG recommeNdation (DESIGN) method that incorporates relevant information embedded in textual and visual modalities of social media posts and models user interests to recommend a plausible set of hashtags. We use word-level attention (WA) on the textual modality followed by a parallel co-attention (PCA) mechanism to model the interaction between textual and visual modalities. Unlike the existing works, we present a hybrid deep neural network that capitalizes hashtags from multilabel classification (MLC) and sequence generation (SG) to recommend candidate hashtags for social media posts. We perform our experiments on social media datasets containing textual, visual, and user information. Experimental results show that the proposed method outperforms the state-of-the-art methods.