Refusal on harmful prompts is a key safety behaviour in instruction‑tuned large language models (LLMs), yet the internal causes of this behaviour remain poorly understood. We study two public instruction tuned models—Gemma‑2-2B‑IT and LLaMA‑3.1-8B‑IT using sparse autoencoders (SAEs) trained on residual‑stream activations. Given a harmful prompt, we search the SAE latent space for feature sets whose ablation flips the model from refusal to compliance, demonstrating causal influence and creating a jailbreak. Our search proceeds in three stages: 1. Refusal Direction - Finding a refusal mediating direction and collecting SAE features close to that direction, followed by 2. Greedy Filtering - to prune this set to obtain a minimal set and finally 3. Interaction Discovery - a factorization‑machine (FM) model that captures non‑linear interactions among the remaining active features and the minimal set. This pipeline yields a broad set of jailbreak-critical features, offering insight into the mechanistic basis of refusal. Moreover, we also find evidence of redundant features which remain dormant unless earlier features are suppressed. Our findings highlight the potential for fine-grained auditing and targeted intervention in safety behaviours by manipulating the interpretable latent space.

Accurate detection of offensive content on social media demands high-quality labeled data; however, such data is often scarce due to the low prevalence of offensive instances and the high cost of manual annotation. To address this low-resource challenge, we propose a self-training framework that leverages abundant unlabeled data through collaborative pseudo-labeling. Starting with a lightweight classifier trained on limited labeled data, our method iteratively assigns pseudo-labels to unlabeled instances with the support of Multi-Agent Vision-Language Models (MA-VLMs). Unlabeled data on which the classifier and MA-VLMs agree are designated as the Agreed-Unknown set, while conflicting samples form the Disagreed-Unknown set. To enhance label reliability, MA-VLMs simulate dual perspectives, moderator and user, capturing both regulatory and subjective viewpoints. The classifier is optimized using a novel Positive-Negative-Unlabeled (PNU) loss, which jointly exploits labeled, Agreed-Unknown, and Disagreed-Unknown data while mitigating pseudo-label noise. Experiments on benchmark datasets demonstrate that our framework substantially outperforms baselines under limited supervision and approaches the performance of large-scale models.

We present SafeLens, a lightweight segment-level video moderation system that fuses speech, text, and visual frames to produce hateful content detection for each segment. For every segment, SafeLens returns a structured prediction: label, prediction confidence, reasons for flag, harm categories. The structured predictions are optimized for triage, appeals, and downstream enforcement. The system is modular (pluggable speech, text, and visual processing modules back-ends and a mid-size policy Language Language Model (LLM) agent with parameter-efficient tuning). In the live demo, attendees can upload or select clips, scrub the timeline to flag hateful segments, inspect rationales, and vary the policy LLM agent to benchmark the hateful content moderation performance.

Large language models (LLMs) have shown impressive multilingual capabilities through pretraining on diverse corpora. While these models show strong reasoning abilities, their performance varies significantly across languages due to imbalanced training data distribution. Existing approaches using sample-level translation for extensive multilingual pretraining and cross-lingual tuning face scalability challenges and often fail to capture nuanced reasoning processes across languages. In this paper, we introduce **AdaMCoT** (Adaptive Multilingual Chain-of-Thought), a framework that enhances multilingual factual reasoning by dynamically routing thought processes in intermediary “thinking languages” before generating target-language responses. AdaMCoT leverages a language-agnostic core and incorporates an adaptive, reward-based mechanism for selecting optimal reasoning pathways without requiring additional pretraining. Our comprehensive evaluation across multiple benchmarks demonstrates substantial improvements in both factual reasoning quality and cross-lingual consistency, with particularly strong performance gains in low-resource language settings. An in-depth analysis of the model’s hidden states and semantic space further elucidates the underlying mechanism of our method. The results suggest that adaptive reasoning paths can effectively bridge the performance gap between high- and low-resource languages while maintaining cultural and linguistic nuances.

Bryan Chen Zhengyu Tan, Zhengyuan Liu, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Roy Ka-Wei Lee. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu, Xiaoxue Gao, Bryan Chen Zhengyu Tan, Bowei Zou, Chang Liu, Yujia Hu, Xing Xie, Xiaoyuan Yi, Jing Yao, Chaojun Wang, Long Li, Rui Liu, Huiyao Liu, Koji Inoue, Ryuichi Sumida, Tatsuya Kawahara, Fan Xu, Lingyu Ye, Wei Tian, Dongjun Kim, Jimin Jung, Jaehyung Seo, Nadya Yuki Wangsajaya, Pham Minh Duc, Ojasva Saxena, Palash Nandi, Xiyan Tao, Wiwik Karlina, Tuan Luong, Keertana Arun Vasan, Roy Ka-Wei Lee, Nancy F. Chen. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Hateful videos have become prevalent on online platforms, highlighting an urgent need for effective detection. However, existing studies primarily focus on binary classification and fail to provide contextual rationales that reveal the implicit meanings behind these judgments, significantly undermining model explainability. To fill this gap, we aim to achieve explainable hateful video detection, enabling models to provide contextual rationales that integrate relevant evidence and logical reasoning alongside decisions. This approach can comprehensively enhance the understanding of video content and the explainability of the decision-making process. We first introduce two datasets, Ex-HateMM and Ex-ImpliHateVid, for explainable hateful video detection. Each dataset provides fine-grained annotations of multimodal harmful elements, along with contextual rationales. We then propose an Information Augmentation and Reasoning Enhancement (IARE) framework designed for explainable detection. The framework employs an information augmentation phase that leverages the multimodal chain-of-thought to integrate harmful elements, thereby enriching rationale evidence. Additionally, IARE incorporates a reasoning enhancement phase, in which Direct Preference Optimization guides the model toward correct reasoning paths and away from incorrect ones, thereby improving the logical coherence of its justifications. We conduct extensive experiments on the two datasets, comparing multiple baselines with our proposed IARE framework. The results demonstrate that IARE achieves state-of-the-art performance while also generating accurate rationales.

Smartphone sensing offers a scalable, unobtrusive window into daily behavioral patterns linked to mental health. While prior work focuses on detection — identifying current conditions — forecasting future mental health states enables proactive, Just-in-Time Adaptive Interventions before crises emerge. We present the first benchmarking study comparing traditional machine learning (ML), deep learning (DL), and large language models (LLMs) for mental health forecasting using the College Experience Sensing dataset — the most extensive longitudinal passive sensing dataset on college student mental health to date. Our evaluation spans modeling paradigm, LLM adaptation strategy, and personalization, yielding three key findings: DL models, particularly Transformer, achieve the strongest overall performance, while LLMs exhibit a reasoning-prediction gap; few-shot consistently outperforms zero-shot and supervised fine-tuning (SFT); and personalization substantially improves forecasting. These findings lay groundwork for next-generation, human-centered adaptive systems supporting early mental health intervention. Our code is available at https://github.com/KaiDF/Study_MentalHealth_Forecasting.

Large Language Models (LLMs) are rapidly transforming recommender systems (RSs) and user modeling by enabling richer representations of users, context, and intent. In wellbeing-oriented applications, such as activity, diet, stress, and mental health support, the use of LLM-based RSs introduces both new opportunities and critical challenges related to trustworthiness, adaptivity, explainability, privacy, and human-centered evaluation. This workshop aims to bring together researchers and industry practitioners from user modeling, personalization, RSs and healthcare to examine how LLM-based models can be responsibly designed, evaluated, and deployed for mental and physical wellbeing, fostering AI for social good. The workshop focuses on adaptive LLM-based user modeling, trustworthy recommendation mechanisms, LLM-powered user simulation, and evaluation methodologies that capture longitudinal and affective user outcomes. Strongly aligned with the ACM UMAP community, the workshop emphasizes adaptive and personalized systems that place human values at the center. Through paper presentations, invited talks, and interactive discussions, the workshop will surface open challenges, emerging best practices, and future research directions for LLM-driven wellbeing recommendation.

Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense, a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision-Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub.

Multilingual Large Language Models (LLMs) struggle with cross-lingual tasks due to data imbalances between high-resource and low-resource languages and the monolingual bias in pre-training. Existing methods, such as bilingual fine-tuning and contrastive alignment, improve cross-lingual performance but often require extensive parallel data or suffer from instability. To address these challenges, we introduce a Cross-Lingual Mapping Task in the pre-training phase, which enhances cross-lingual alignment without compromising monolingual fluency. Our approach bi-directionally maps languages within the LLM’s embedding space, improving both language generation and comprehension. We further introduce a Language Alignment Coefficient to robustly quantify cross-lingual consistency, even in limited-data scenarios. Experimental results on machine translation (MT), cross-lingual natural language understanding (CLNLU), and cross-lingual question answering (CLQA) show that our model achieves up to 11.9 BLEU score gains in MT, an increase of 6.72 in CLQA BERTScore-Precision and more than a 5% increase in CLNLU accuracy over strong multilingual baselines. Our findings highlight the potential of embedding cross-lingual objectives into pre-training, improving multilingual LLMs.

The widespread use of social media has accelerated the dissemination of information, but it has also facilitated the spread of harmful rumors, which can disrupt economies, influence political outcomes, and exacerbate public health crises, such as the COVID-19 pandemic. While graph neural network (GNN)- based approaches have shown significant promise in automated rumor detection, they often lack transparency, making their predictions difficult to interpret. Existing graph explainability techniques fall short in addressing the unique challenges posed by the dependencies among feature dimensions in high-dimensional text embeddings used in GNN-based models. In this article, we introduce contrastive token layerwise relevance propagation (CTLRP), a novel framework designed to enhance the explainability of GNN-based rumor detection. CT-LRP extends current graph explainability methods by providing token-level explanations that offer greater granularity and interpretability. We evaluate the effectiveness of CT-LRP across multiple GNN models trained on three publicly available rumor detection datasets, demonstrating that it consistently produces high-fidelity, meaningful explanations, paving the way for more robust and trustworthy rumor detection systems.

Social media platforms have democratised information creation and dissemination by empowering users worldwide to reach vast audiences almost instantaneously. However, these platforms have also become vectors for spreading misinformation such as rumours. If left unchecked, rumours have the potential to cause great economic and political damage and even worsen public health crises. Automated rumour detection systems are imperative to deal with the volume and velocity of information being exchanged on these platforms. Graph Neural Network (GNN)-based approaches have recently emerged as state-of-the-art (SOTA) in automated rumour detection. Despite their performance, these models remain largely opaque, making explaining their predictions challenging, particularly when dealing with noisy social media data. Existing graph explainability techniques struggle to produce high-fidelity contrastive explanations when presented with such noisy data especially when dealing with a multiclass classification problem. To address the issue of noise susceptibility, we propose a novel framework to maximise both contrastivity and fidelity by reframing the explanation task as a maximum margin optimisation problem. Specifically, we impose constraints on explanation set membership and on the influence difference of the prediction explanation set to each other class explanation set. Extensive experiments on real-world datasets show that the proposed method outperforms SOTA methods.

Yao Xiao, Hai Ye, Linyao Chen, Hwee Tou Ng, Lidong Bing, Xiaoli Li, Roy Ka-Wei Lee. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Social recommendation, which leverages users' social information to predict users' preferences, is a popular branch of recommender systems. Many existing studies have attempted to advance the performance of collaborative filtering methods by leveraging the user-user matrix to enhance user embedding learning with user's social connections. While the existing social recommender systems have demonstrated good performance in various recommendation tasks, the extent of social information usefulness in recommender systems remains unclear. This paper addresses the research gap by designing experiments to answer three research questions: (i) How useful is social information in varying user-item data sparsity? (ii) How much social information do the existing social recommendation models use? (iii) How valuable is social information for cold-start situations? Working towards answering the research questions, we introduce evaluation metrics to estimate the utilization of social information in the existing social recommendation models. We conducted experiments on three publicly available social recommendation datasets, and our results showed that there are diminishing returns when applying social information in recommender systems.

The advancement of Large Language Models (LLMs) has transformed natural language processing; however, their safety mechanisms remain under-explored in low-resource, multilingual settings.Here, we aim to bridge this gap.In particular, we introduce SGToxic-Guard, a novel dataset and evaluation framework for benchmarking LLM safety in Singapore's diverse linguistic context, including Singlish, Chinese, Malay, and Tamil.SGToxi-cGuard adopts a red-teaming approach to systematically probe LLM vulnerabilities in three real-world scenarios: conversation, questionanswering, and content composition.We conduct extensive experiments with state-of-the-art multilingual LLMs, and the results uncover critical gaps in their safety guardrails.By offering actionable insights into cultural sensitivity and toxicity mitigation, we lay the foundation for safer and more inclusive AI systems in linguistically diverse environments.1 Disclaimer: This paper contains sensitive content that may be disturbing to some readers.

Large Language Models (LLMs) can struggle to balance gullibility to misinformation and resistance to valid corrections in persuasive dialogues, a critical challenge for reliable deployment.We introduce DuET-PD (Dual Evaluation for Trust in Persuasive Dialogues), a framework evaluating multi-turn stancechange dynamics across dual dimensions: persuasion type (corrective/misleading) and domain (knowledge via MMLU-Pro, and safety via SALAD-Bench).We find that even a stateof-the-art model like GPT-4o achieves only 27.32% accuracy in MMLU-Pro under sustained misleading persuasions.Moreover, results reveal a concerning trend of increasing sycophancy in newer open-source models.To address this, we introduce Holistic DPO, a training approach balancing positive and negative persuasion examples.Unlike prompting or resist-only training, Holistic DPO enhances both robustness to misinformation and receptiveness to corrections, improving Llama-3.1-8B-Instruct'saccuracy under misleading persuasion in safety contexts from 4.21% to 76.54%.These contributions offer a pathway to developing more reliable and adaptable LLMs for multi-turn dialogue.Code is available at https://github.com/Social-AI-Studio/DuET-PD.

Current benchmarks like ``$\textit{Needle-in-a-Haystack}$'' ($\textit{NIAH}$), $\textit{Ruler}$, and $\textit{Needlebench}$ focus on models' ability to understand long-context input sequences but fail to capture a critical dimension: the generation of high-quality long-form text. Applications such as design proposals, technical documentation, and creative writing rely on coherent, instruction-following outputs over extended sequences—a challenge that existing benchmarks do not adequately address. To fill this gap, we introduce $\textit{LongGenBench}$, a novel benchmark designed to rigorously evaluate large language models' (LLMs) ability to generate long text while adhering to complex instructions. Through tasks requiring specific events or constraints within generated text, $\textit{LongGenBench}$ evaluates model performance across four distinct scenarios, three instruction types, and two generation-lengths (16K and 32K tokens). Our evaluation of ten state-of-the-art LLMs reveals that, despite strong results on $\textit{Ruler}$, all models struggled with long text generation on $\textit{LongGenBench}$, particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation. We open-source $\textit{LongGenBench}$ to promote comprehensive evaluation and improvement in this critical area, with code and data available at ${anonymousurl}$.

We present a comprehensive empirical study on how backdoor attacks affect CLIP by analyzing the representations of backdoor images. Specifically, based on the methodology of representation decomposing, image representations can be decomposed into a sum of representations across individual image patches, attention heads (AHs), and multi-layer perceptrons (MLPs) in different model layers. By examining the effect of backdoor attacks on model components, we have the following empirical findings. (1) Different backdoor attacks would infect different model components, i.e., local patch-based backdoor attacks mainly affect AHs, while global perturbation-based backdoor attacks mainly affect MLPs. (2) Infected AHs are centered on the last layer, while infected MLPs are decentralized on several late layers. (3) Not all AHs in the last layer are infected and even some AHs could still maintain the original property-specific roles (e.g., ''color" and ''location''). These observations motivate us to defend against backdoor attacks by detecting infected AHs, repairing their representations, or filtering backdoor samples with too many infected AHs, in the inference stage. Experimental results validate our empirical findings and demonstrate the effectiveness of the defense methods.

Detecting hateful memes requires a model that possesses extensive background knowledge and robust reasoning abilities, especially when the memes contain ambiguous descriptions. Previous research has used large language models (LLMs) and large multimodal models (LMMs) to interpret and categorize these memes. However, distinguishing subtly different hateful and non-hateful memes is still challenging. In recognition of this, our study introduces a unique contrastive instruction fine-tuning approach, InstructMemeCL. This method improves an LMM's ability to discern between memes that have similar visual or textual elements by intensifying its focus on semantic subtleties that separate hateful from non-hateful content. We evaluated our model using AUROC and accuracy metrics on three publicly available hateful meme datasets. The results indicate that our improved LMM more accurately identifies hateful and non-hateful memes, demonstrating superior performance compared to conventional LLMs and LMMs used in similar tasks.

Hateful meme detection presents a significant challenge as a multimodal task due to the complexity of interpreting implicit hate messages and contextual cues within memes. Previous approaches have fine-tuned pre-trained vision-language models (PT-VLMs), leveraging the knowledge they gained during pre-training and their attention mechanisms to understand meme content. However, the reliance of these models on implicit knowledge and complex attention mechanisms renders their decisions difficult to explain, which is crucial for building trust in meme classification. In this paper, we introduce IntMeme, a novel framework that leverages Large Multimodal Models (LMMs) for hateful meme classification with explainable decisions. IntMeme addresses the dual challenges of improving both accuracy and explainability in meme moderation. The framework uses LMMs to generate human-like, interpretive analyses of memes, providing deeper insights into multimodal content and context. Additionally, it uses independent encoding modules for both memes and their interpretations, which are then combined to enhance classification performance. Our approach addresses the opacity and misclassification issues associated with PT-VLMs, optimizing the use of LMMs for hateful meme detection. We demonstrate the effectiveness of IntMeme through comprehensive experiments across three datasets, showcasing its superiority over state-of-the-art models.

Large Language Models (LLMs) augmented with retrieval mechanisms have demonstrated significant potential in fact-checking tasks by integrating external knowledge. However, their reliability decreases when confronted with conflicting evidence from sources of varying credibility. This paper presents the first systematic evaluation of Retrieval-Augmented Generation (RAG) models for fact-checking in the presence of conflicting evidence. To support this study, we introduce CONFACT (Conflicting Evidence for Fact-Checking), a novel dataset comprising questions paired with conflicting information from various sources. Extensive experiments reveal critical vulnerabilities in state-of-the-art RAG methods, particularly in resolving conflicts stemming from differences in media source credibility. To address these challenges, we investigate strategies to integrate media background information into both the retrieval and generation stages. Our results show that effectively incorporating source credibility significantly enhances the ability of RAG models to resolve conflicting evidence and improve fact-checking performance.

Detecting hate speech in videos remains challenging due to the complexity of multimodal content and the lack of fine-grained annotations in existing datasets. We present HateClipSeg, a large-scale multimodal dataset with both video-level and segment-level annotations, comprising over 11,714 segments labeled as Normal or across five Offensive categories: Hateful, Insulting, Sexual, Violence, Self-Harm, along with explicit target victim labels. Our three-stage annotation process yields high inter-annotator agreement (Krippendorff's alpha = 0.817). We propose three tasks to benchmark performance: (1) Trimmed Hateful Video Classification, (2) Temporal Hateful Video Localization, and (3) Online Hateful Video Classification. Results highlight substantial gaps in current models, emphasizing the need for more sophisticated multimodal and temporally aware approaches. The HateClipSeg dataset are publicly available at https://github.com/Social-AI-Studio/HateClipSeg.git.

Bryan Chen Zhengyu Tan, Roy Ka-Wei Lee. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Detecting hate speech in online content is essential to ensuring safer digital spaces. While significant progress has been made in text and meme modalities, video-based hate speech detection remains under-explored, hindered by a lack of annotated datasets and the high cost of video annotation. This gap is particularly problematic given the growing reliance on large models, which demand substantial amounts of training data. To address this challenge, we leverage meme datasets as both a substitution and an augmentation strategy for training hateful video detection models. Our approach introduces a human-assisted reannotation pipeline to align meme dataset labels with video datasets, ensuring consistency with minimal labeling effort. Using two state-of-the-art vision-language models, we demonstrate that meme data can substitute for video data in resource-scarce scenarios and augment video datasets to achieve further performance gains. Our results consistently outperform state-of-the-art benchmarks, showcasing the potential of cross-modal transfer learning for advancing hateful video detection. Dataset and code are available at (https://github.com/Social-AI-Studio/CrossModalTransferLearning).

Task-oriented dialogue (TOD) systems play a critical role in assisting users with various tasks, such as ticket booking and service inquiries. While these systems have demonstrated significant potential in addressing customer needs, they typically assume that users will interact with the dialogue agent in a polite manner. This assumption, however, is often unrealistic, as users may express impatience or frustration through impolite behavior. Addressing this gap, this article investigates the impact of impolite user behavior on the performance of TOD systems. To this end, we developed a novel corpus of impolite dialogues and conducted comprehensive experiments to evaluate the performance of state-of-the-art TOD systems on this dataset. Our results reveal a notable limitation: existing TOD systems struggle to handle impolite user utterances effectively, leading to degraded performance. To mitigate this issue, we introduce a data augmentation approach designed to improve the systems’ ability to manage impolite dialogues. Although this method achieves measurable improvements, managing impolite user interactions remains a challenging research problem. By making our impolite dialogue corpus publicly accessible, we aim to encourage further research in this underexplored area. This study underscores the need for more robust TOD systems capable of handling diverse user behaviors, ultimately enhancing their applicability in real-world scenarios.

The widespread presence of hate speech on the internet, including formats such as text-based tweets and vision-language memes, poses a significant challenge to digital platform safety.Recent research has developed detection models tailored to specific modalities; however, there is a notable gap in transferring detection capabilities across different formats.This study conducts extensive experiments using few-shot in-context learning with large language models to explore the transferability of hate speech detection between modalities.Our findings demonstrate that text-based hate speech examples can significantly enhance the classification accuracy of vision-language hate speech.Moreover, text-based demonstrations outperform vision-language demonstrations in fewshot learning settings.These results highlight the effectiveness of cross-modality knowledge transfer and offer valuable insights for improving hate speech detection systems 1 .* These authors contributed equally to this work. 1GitHub: https://github.com/Social-AI-Studio/ Bridging-Modalities is complicated by copyright issues and increasingly stringent regulations on social platforms.Consequently, the limited availability of vision-language data hampers performance in out-of-distribution cases.In contrast, the abundance and diversity of text-based data offer a potential source for crossmodality knowledge transfer (Hee et al., 2024).Research Objectives.This paper investigates whether text-based hate speech detection capabilities can be transferred to multimodal formats.By leveraging the richness of text-based data, we aim to enhance the detection of vision-language hate speech, addressing current research limitations and improving performance in low-resource settings.Contributions.This study makes the following key contributions: (i) We conduct extensive experiments evaluating the transferability of textbased hate speech detection to vision-language formats using few-shot in-context learning with large language models.(ii) We demonstrate that textbased hate speech examples significantly improve the classification accuracy of vision-language hate speech.(iii) We show that text-based demonstrations in few-shot learning contexts outperform vision-language demonstrations, highlighting the potential for cross-modality knowledge transfer.These contributions address critical gaps in existing research and provide a foundation for developing robust hate speech detection systems. Research QuestionsAs all forms of hate speech share one definition, this study investigates the usefulness of using hate speech from one form, such as text-based hate speech, to classify hate speech in another form, such as vision-language hate speech.Working to-

Detecting hate speech and offensive language is essential for maintaining a safe and respectful digital environment.This study examines the limitations of state-of-the-art large language models (LLMs) in identifying offensive content within systematically perturbed data, with a focus on Chinese, a language particularly susceptible to such perturbations.We introduce ToxiCloakCN 1 , an enhanced dataset derived from ToxiCN, augmented with homophonic substitutions and emoji transformations, to test the robustness of LLMs against these cloaking perturbations.Our findings reveal that existing models significantly underperform in detecting offensive content when these perturbations are applied.We provide an in-depth analysis of how different types of offensive content are affected by these perturbations and explore the alignment between human and model explanations of offensiveness.Our work highlights the urgent need for more advanced techniques in offensive language detection to combat the evolving tactics used to evade detection mechanisms.

Hate speech is a pressing issue in modern society, with significant effects both online and offline. Recent research in hate speech detection has primarily centered on text-based media, largely overlooking multimodal content such as videos. Existing studies on hateful video datasets have predominantly focused on English content within a Western context and have been limited to binary labels (hateful or non-hateful), lacking detailed contextual information. This study presents MultiHateClip, an novel multilingual dataset created through hate lexicons and human annotation. It aims to enhance the detection of hateful videos on platforms such as YouTube and Bilibili, including content in both English and Chinese languages. Comprising 2,000 videos annotated for hatefulness, offensiveness, and normalcy, this dataset provides a cross-cultural perspective on gender-based hate speech. Through a detailed examination of human annotation results, we discuss the differences between Chinese and English hateful videos and underscore the importance of different modalities in hateful and offensive video analysis. Evaluations of state-of-the-art video classification models, such as VLM, GPT-4V and Qwen-VL, on MultiHateClip highlight the existing challenges in accurately distinguishing between hateful and offensive content and the urgent need for models that are both multimodally and culturally nuanced. MultiHateClip represents a foundational advance in enhancing hateful video detection by underscoring the necessity of a multimodal and culturally sensitive approach in combating online hate speech.

In this paper, we address the challenge of detecting hateful memes in the low-resource setting where only a few labeled examples are available. Our approach leverages the compositionality of Low-rank adaptation (LoRA), a widely used parameter-efficient tuning technique. We commence by fine-tuning large language models (LLMs) with LoRA on selected tasks pertinent to hateful meme detection, thereby generating a suite of LoRA modules. These modules are capable of essential reasoning skills for hateful meme detection. We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.

Online memes have emerged as powerful digital cultural artifacts in the age of social media, offering not only humor but also platforms for political discourse, social critique, and information dissemination. Their extensive reach and influence in shaping online communities' sentiments make them invaluable tools for campaigning and promoting ideologies. Despite the development of several meme generation tools, there remains a gap in their systematic evaluation and their ability to effectively communicate ideologies. Addressing this, we introduce MemeCraft, an innovative meme generator that leverages large language models (LLMs) and visual language models (VLMs) to produce memes advocating specific social movements. MemeCraft presents an end-to-end pipeline, transforming user prompts into compelling multimodal memes without manual intervention. Conscious of the misuse potential in creating divisive content, an intrinsic safety mechanism is embedded to curb hateful meme production. Our assessment, focusing on two UN Sustainable Development Goals-Climate Action and Gender Equality-shows MemeCraft's prowess in creating memes that are both funny and supportive of advocacy goals. This paper highlights how generative AI can promote social good and pioneers the use of LLMs and VLMs in meme generation.

Hate speech in social media is a growing phenomenon, and detecting such toxic content has recently gained significant traction in the research community. Existing studies have explored fine-tuning language models (LMs) to perform hate speech detection, and these solutions have yielded significant performance. However, most of these studies are limited to detecting hate speech only in English, neglecting the bulk of hateful content that is generated in other languages, particularly in low-resource languages. Developing a classifier that captures hate speech and nuances in a low-resource language with limited data is extremely challenging. To fill the research gap, we proposeHateMAML, a model-agnostic meta-learning (MAML)-based framework that effectively performs hate speech detection in low-resource languages.HateMAMLutilizes a self-supervision strategy to overcome the limitation of data scarcity and produces better LM initialization for fast adaptation to an unseen target language (i.e., cross-lingual transfer) or other hate speech datasets (i.e., domain generalization). Extensive experiments are conducted on five datasets across eight different low-resource languages. The results show thatHateMAMLoutperforms the state-of-the-art baselines by more than 3% in the cross-domain multilingual transfer setting. We also conduct ablation studies to analyze the characteristics ofHateMAML.

The rise of social media has enabled individuals to rapidly share information, including rumors, which can have significant impacts on various domains. Traditional approaches to rumor control are impractical for social media platforms due to the volume and speed of information. Automated detection methods are needed that not only identify rumors early but also provide explanations for their decisions to protect free speech. Recent advancements in deep learning have shown promise in automating rumor detection. Graph-based models, such as bidirectional graph convolution network (Bi-GCN), capture propagation, and dispersion patterns to differentiate rumors from the truth. However, the interpretability of these deep learning models is a challenge. This article focuses on graph convolution networks (GCNs), which lack attention maps for easy model attribution but excel at capturing global structural features. We investigate the importance of graph structure in rumor detection using two GCN models on a real-world dataset, analyzing the learned latent propagation and dispersion features. To the best of our knowledge, this is the first study to explore GCNs in rumor detection and investigate the significance of graph structure in this task. Our research addresses three primary questions: 1) the primary contributors to GCN-based rumor detection models and their differences across models; 2) the importance of graph structure for accurate predictions in GCN-based models; and 3) the latent propagation and dispersion features learned by GCN-based detection models during the rumor detection process.

Online rumor detection is crucial for a healthier online environment. Traditional methods mainly rely on content understanding. However, these contents can be easily adjusted to avoid such supervision and are insufficient to improve the detection result. Compared with the content, information propagation patterns are more informative to support further performance promotion. Unfortunately, learning the propagation patterns is difficult, since the retweeting tree is more topologically complicated than linear sequences or binary trees. In light of this, we propose a novel rumor detection framework based on structure-aware retweeting graph neural networks. To capture the propagation patterns, we first design a novel conversion method to transform the complex retweeting tree as more tractable binary tree without losing the reconstruction information. Then, we serialize the retweeting tree as a corpus of meta-tree paths, where each meta-tree can preserve a basic substructure. A deep neural network is then designed to integrate all meta-trees and to generate the global structural embeddings. Furthermore, we propose to integrate content, users, and propagation patterns to enhance more reliable performance. To this end, we propose a novel self-attention-based retweeting neural network to learn individual features from both content and users. We then fuse the node-level features with our global structural embeddings via a mutual attention unit. In this way, we can generate more comprehensive representations for rumor detection. Extensive evaluations on two real-world datasets show remarkable superiorities of our model compared with existing methods.

While AI-assisted individual qualitative analysis has been substantially studied, AI-assisted collaborative qualitative analysis (CQA) – a process that involves multiple researchers working together to interpret data—remains relatively unexplored. After identifying CQA practices and design opportunities through formative interviews, we designed and implemented CoAIcoder, a tool leveraging AI to enhance human-to-human collaboration within CQA through four distinct collaboration methods. With a between-subject design, we evaluated CoAIcoder with 32 pairs of CQA-trained participants across common CQA phases under each collaboration method. Our findings suggest that while using a shared AI model as a mediator among coders could improve CQA efficiency and foster agreement more quickly in the early coding stage, it might affect the final code diversity. We also emphasize the need to consider the independence level when using AI to assist human-to-human collaboration in various CQA scenarios. Lastly, we suggest design implications for future AI-assisted CQA systems.

Janosch Haber, Bertie Vidgen, Matthew Chapman, Vibhor Agarwal, Roy Ka-Wei Lee, Yong Keong Yap, Paul Röttger. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

When traveling to an unfamiliar city for holidays, tourists often rely on guidebooks, travel websites, or recommendation systems to plan their daily itineraries and explore popular points of interest (POIS). However, these approaches may lack optimization in terms of time feasibility, localities, and user preferences. In this paper, we propose the SBTREC algorithm: a BERT-based Trajectory Recommendation with sentiment analysis, for recommending personalized sequences of POIS as itineraries. Considering the locations, sightseeing, and travel time between consecutive Pots, our approach incorporates individual user preferences through the utilization of historical data. The key contributions of this work include analyzing users’ check-ins and uploaded photos to understand the relationship between Pot visits and distance. In addition, SBTREC also encompasses sentiment analysis to improve recommendation accuracy by understanding users’ preferences and satisfaction levels from reviews and comments about different Pots. Our proposed algorithms are evaluated against other sequence prediction methods using datasets from 8 cities. The results demonstrate that SBTREC achieves an average $\mathcal{F}_{1}$ score of 61.45%, outperforming baseline algorithms. The paper further discusses the flexibility of the SBTREC algorithm, its ability to adapt to different scenarios and cities without modification, and its potential for extension by incorporating additional information for more reliable predictions. Overall, SBTREC provides personalized and relevant Pot recommendations, enhancing tourists’ overall trip experiences. Future work includes fine-tuning personalized embeddings for users, with evaluation of users’ comments on Pots, to further enhance prediction accuracy.

The intentional spread of misinformation can have serious consequences in our society. This motivates us to address the task of identifying sock puppet accounts (i.e. fabricated online personas) created by individuals or organizations with the intention of deceiving their target audience. By approaching the problem as authorship attribution, we develop a sock puppet detection framework that relies solely on the texts posted by sock puppets without using any meta-information. We employed a large pre-trained language model and used interpretability methods to extract linguistic cues to answer our research question. We curated a high-quality sock puppet dataset to enable a comprehensive study for the research communities where existing datasets may include false positives due to their approximate approaches in identifying sock puppets. This dataset enables us to study the research question of whether authorship attribution works on sock puppets’ written texts. Our experiment shows that our method remarkably outperformed human annotators by 31.8% points. Furthermore, we employed an interpretability method, Integrated Gradient, to extract linguistic cues from our model for explanations of why an account is a sock puppet or not.

Memes are widely used to convey cultural and societal issues and have a significant impact on public opinion. However, little work has been done on understanding and explaining the semantics expressed in multimodal memes. To fill this research gap, we introduce MERMAID, a dataset consisting of 3,633 memes annotated with their entities and relations, and propose a novel MERF pipeline that extracts entities and their relationships in memes. Our framework combines state-of-the-art techniques from natural language processing and computer vision to extract text and image features and infer relationships between entities in memes. We evaluate the proposed framework on a real-world meme dataset and establish the benchmark for the new multimodal meme semantic understanding task. Our evaluation also includes a low-resource setting, where we assess the applicability of our framework to low-resource settings, which is a common problem due to the high cost and lack of labeled data for relations in memes. Overall, our work contributes to the understanding of the semantics of memes, a crucial form of communication in today’s society.

Topic modeling is a widely used technique for revealing underlying thematic structures within textual data. However, existing models have certain limitations, particularly when dealing with short text datasets that lack co-occurring words. Moreover, these models often neglect sentence-level semantics, focusing primarily on token-level semantics. In this paper, we propose PromptTopic, a novel topic modeling approach that harnesses the advanced language understanding of large language models (LLMs) to address these challenges. It involves extracting topics at the sentence level from individual documents, then aggregating and condensing these topics into a predefined quantity, ultimately providing coherent topics for texts of varying lengths. This approach eliminates the need for manual parameter tuning and improves the quality of extracted topics. We benchmark PromptTopic against the state-of-the-art baselines on three vastly diverse datasets, establishing its proficiency in discovering meaningful topics. Furthermore, qualitative analysis showcases PromptTopic’s ability to uncover relevant topics in multiple datasets.

The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca).Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance.To enable further research on PEFT methods of LLMs, this paper presents LLM-Adapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks.The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods.Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods.We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning.The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.The code and datasets can be found in https://github. com/AGI-Edgerunners/LLM-Adapters.

Recent studies have proposed models that yielded promising performance for the hateful meme classification task. Nevertheless, these proposed models do not generate interpretable explanations that uncover the underlying meaning and support the classification output. A major reason for the lack of explainable hateful meme methods is the absence of a hateful meme dataset that contains ground truth explanations for benchmarking or training. Intuitively, having such explanations can educate and assist content moderators in interpreting and removing flagged hateful memes. This paper address this research gap by introducing Hateful meme with Reasons Dataset (HatReD), which is a new multimodal hateful meme dataset annotated with the underlying hateful contextual reasons. We also define a new conditional generation task that aims to automatically generate underlying reasons to explain hateful memes and establish the baseline performance of state-of-the-art pre-trained language models on this task. We further demonstrate the usefulness of HatReD by analyzing the challenges of the new conditional generation task in explaining memes in seen and unseen domains. The dataset and benchmark models are made available here: https://github.com/Social-AI-Studio/HatRed

Recent research has focused on using large language models (LLMs) to generate explanations for hate speech through fine-tuning or prompting. Despite the growing interest in this area, these generated explanations' effectiveness and potential limitations remain poorly understood. A key concern is that these explanations, generated by LLMs, may lead to erroneous judgments about the nature of flagged content by both users and content moderators. For instance, an LLM-generated explanation might inaccurately convince a content moderator that a benign piece of content is hateful. In light of this, we propose an analytical framework for examining hate speech explanations and conducted an extensive survey on evaluating such explanations. Specifically, we prompted GPT-3 to generate explanations for both hateful and non-hateful content, and a survey was conducted with 2,400 unique respondents to evaluate the generated explanations. Our findings reveal that (1) human evaluators rated the GPT-generated explanations as high quality in terms of linguistic fluency, informativeness, persuasiveness, and logical soundness, (2) the persuasive nature of these explanations, however, varied depending on the prompting strategy employed, and (3) this persuasiveness may result in incorrect judgments about the hatefulness of the content. Our study underscores the need for caution in applying LLM-generated explanations for content moderation. Code and results are available at https://github.com/Social-AI-Studio/GPT3-HateEval.

Destination prediction based on the partial trajectory of a moving vehicle is vital for urban mobility applications. Recent research efforts focus on improving the prediction accuracy by incorporating more spatio-temporal semantics through complex model architectures, which inevitably impact the generalization and scalability due to ad-hoc hyper-parameters and heavier computations. In the present study, we propose a novel Location Recommendation System for Destination Prediction, LRS4DP. Through an integrated design of several technologies (map-matching, deep learning and recommender system), LRS4DP provides an end-to-end solution for destination prediction based on input trajectories and road network configurations. By adopting a node-based spatial discretization scheme through map-matching, LRS4DP is able to adapt according to the local road network density and generalize to different urban layouts. As compared to the state-of-the-art algorithms, our proposed Top-K formulation based on individual road nodes leads to fundamentally better spatial precision and prediction accuracy even with simple model architectures. We further designed the offline training and online serving as a location recommendation system to achieve better scalability and flexible trade-off between performance and run-time. The experimental evaluation of two real-world taxi datasets demonstrates the generalization of LRS4DP under different urban scales and layouts. The LRS4DP framework is also generically applicable for location prediction tasks (e.g., next location and passing-by location predictions) and capable to support various downstream transportation and location-based service applications.

Hateful meme detection is a challenging multimodal task that requires comprehension of both vision and language, as well as cross-modal interactions. Recent studies have tried to fine-tune pre-trained vision-language models (PVLMs) for this task. However, with increasing model sizes, it becomes important to leverage powerful PVLMs more efficiently, rather than simply fine-tuning them. Recently, researchers have attempted to convert meme images into textual captions and prompt language models for predictions. This approach has shown good performance but suffers from non-informative image captions. Considering the two factors mentioned above, we propose a probing-based captioning approach to leverage PVLMs in a zero-shot visual question answering (VQA) manner. Specifically, we prompt a frozen PVLM by asking hateful content-related questions and use the answers as image captions (which we call Pro-Cap), so that the captions contain information critical for hateful content detection. The good performance of models with Pro-Cap on three benchmarks validates the effectiveness and generalization of the proposed method1.

The rise of social media platforms has brought about a new digital culture called memes. Memes, which combine visuals and text, can strongly influence public opinions on social and cultural issues. As a result, people have become interested in categorizing memes, leading to the development of various datasets and multimodal models that show promising results in this field. However, there is currently a lack of a single library that allows for the reproduction, evaluation, and comparison of these models using fair benchmarks and settings. To fill this gap, we introduce the Meme Analytical Tool Kit (MATK), an open-source toolkit specifically designed to support existing memes datasets and cutting-edge multimodal models. MATK aims to assist researchers and engineers in training and reproducing these multimodal models for meme classification tasks, while also providing analysis techniques to gain insights into their strengths and weaknesses. To access MATK, please visit https://github.com/Social-AI-Studio/MATK.

The proliferation of social media has given rise to a new form of communication: memes. Memes are multimodal and often contain a combination of text and visual elements that convey meaning, humor, and cultural significance. While meme analysis has been an active area of research, little work has been done on unsupervised multimodal topic modeling of memes, which is important for content moderation, social media analysis, and cultural studies. We propose PromptMTopic, a novel multimodal prompt-based model designed to learn topics from both text and visual modalities by leveraging the language modeling capabilities of large language models. Our model effectively extracts and clusters topics learned from memes, considering the semantic interaction between the text and visual modalities. We evaluate our proposed model through extensive experiments on three real-world meme datasets, which demonstrate its superiority over state-of-the-art topic modeling baselines in learning descriptive topics in memes. Additionally, our qualitative analysis shows that PromptMTopic can identify meaningful and culturally relevant topics from memes. Our work contributes to the understanding of the topics and themes of memes, a crucial form of communication in today's society. Disclaimer: This paper contains sensitive content that may be disturbing to some readers.

Total Defence is a defence policy combining and extending the concept of military defence and civil defence. While several countries have adopted total defence as their defence policy, very few studies have investigated its effectiveness. With the rapid proliferation of social media and digitalisation, many social studies have been focused on investigating policy effectiveness through specially curated surveys and questionnaires either through digital media or traditional forms. However, such references may not truly reflect the underlying sentiments about the target policies or initiatives of interest. People are more likely to express their sentiment using communication mediums such as starting topic thread on forums or sharing memes on social media. Using Singapore as a case reference, this study aims to address this research gap by proposing TotalDefMeme, a large-scale multi-modal and multi-attribute meme dataset that captures public sentiments toward Singapore's Total Defence policy. Besides supporting social informatics and public policy analysis of the Total Defence policy, TotalDefMeme can also support many downstream multi-modal machine learning tasks, such as aspect-based stance classification and multi-modal meme clustering. We perform baseline machine learning experiments on TotalDefMeme and evaluate its technical validity, and present possible future interdisciplinary research directions and application scenarios using the dataset as a baseline.

Clickbait is a sort of news that contains low-quality content but intriguing or overstated titles. Most of the existing studies usually utilize titles to detect clickbait and suffer from poor performance. However, from the perspective of user cognition, the essence of clickbait news is due to the fact that the body contents fall short of the titles’ expectations. So, in addition to the explicit features of titles, many extra informative signals could help to distinguish clickbait. In this article, we proposed a novel Multiview learning approach-based ClickBait Detection (MCBD) model that takes into account useful signals from both titles and body contents. From the view of titles, a multilayer gated convolutional network is employed to learn the local- and long-distance dependence relationships between words. From the view of body contents, we first extract the key topic sentences from the body content and then we developed a novel attention-fused deep relevance matching network (ARMN) to thoroughly mine the similarity between titles and body contents. Combined with the information from the views of titles and body contents, our model can distinguish well-hidden clickbait news. Finally, extensive experiments were conducted on two real-world datasets to demonstrate that the MCBD model outperforms the state-of-the-art models, improving the final performance by 3% in terms of the F1-score. The ablation studies demonstrate that the relevancy between titles and body contents can improve the model’s performance by 1.7% on the F1-score.

The rapid spread of rumors on social media and their potential impact has motivated the development of automatic rumor detection solutions. However, the existing solutions are mostly limited to detecting rumors in English which neglects the bulk of social media content in other low-resource languages. This paper aims to address the research gaps by proposing Multilingual Source Co-Attention Transformer (MUSCAT), which builds on a multilingual pre-trained language model to perform multilingual rumor detection. Specifically, MUSCAT pivots the source claims in multilingual conversation threads with co-attention transformers to improve detection performance in multilingual settings. We additionally construct multilingual rumor datasets to support our experimental evaluations. Our experimental results show that MUSCAT outperforms state-of-the-art methods in monolingual, cross-lingual, and multilingual rumor detection settings. We have also conducted empirical analysis and outlined the challenges of performing rumor detection in multilingual and cross-lingual settings.

Hateful meme classification is a challenging multimodal task that requires complex reasoning and contextual background knowledge.Ideally, we could leverage an explicit external knowledge base to supplement contextual and cultural information in hateful memes.However, there is no known explicit external knowledge base that could provide such hate speech contextual information.To address this gap, we propose PromptHate, a simple yet effective prompt-based model that prompts pre-trained language models (PLMs) for hateful meme classification.Specifically, we construct simple prompts and provide a few in-context examples to exploit the implicit knowledge in the pretrained RoBERTa language model for hateful meme classification.We conduct extensive experiments on two publicly available hateful and offensive meme datasets.Our experimental results show that PromptHate is able to achieve a high AUC of 90.96, outperforming state-ofthe-art baselines on the hateful meme classification task.We also perform fine-grained analyses and case studies on various prompt settings and demonstrate the effectiveness of the prompts on hateful meme classification.

Hateful meme detection is a new multimodal task that has gained significant traction in academic and industry research communities. Recently, researchers have applied pre-trained visual-linguistic models to perform the multimodal classification task, and some of these solutions have yielded promising results. However, what these visual-linguistic models learn for the hateful meme classification task remains unclear. For instance, it is unclear if these models are able to capture the derogatory or slurs references in multimodality (i.e., image and text) of the hateful memes. To fill this research gap, this paper propose three research questions to improve our understanding of these visual-linguistic models performing the hateful meme classification task. We found that the image modality contributes more to the hateful meme classification task, and the visual-linguistic models are able to perform visual-text slurs grounding to a certain extent. Our error analysis also shows that the visual-linguistic models have acquired biases, which resulted in false-positive predictions.

TikTok is a popular new social media, where users express themselves through short video clips. A common form of interaction on the platform is participating in "challenges", which are songs and dances for users to iterate upon. Challenge contagion can be measured through replication reach, i.e., users uploading videos of their participation in the challenges. The uniqueness of the TikTok platform where both challenge content and user preferences are evolving requires the combination of challenge and user representation. This paper investigates social contagion of TikTok challenges through predicting a user's participation. We propose a novel deep learning model, deepChallenger, to learn and combine latent user and challenge representations from past videos to perform this user-challenge prediction task. We collect a dataset of over 7,000 videos from 12 trending challenges on the ForYouPage, the app's landing page, and over 10,000 videos from 1303 users. Extensive experiments are conducted and the results show that our proposed deepChallenger (F1=0.494) outperforms baselines (F1=0.188) in the prediction task.

Hateful and offensive content detection has been extensively explored in a single modality such as text. However, such toxic information could also be communicated via multimodal content such as online memes. Therefore, detecting multimodal hateful content has recently garnered much attention in academic and industry research communities. This paper aims to contribute to this emerging research topic by proposing DisMultiHate, which is a novel framework that performed the classification of multimodal hateful content. Specifically, DisMultiHate is designed to disentangle target entities in multimodal memes to improve the hateful content classification and explainability. We conduct extensive experiments on two publicly available hateful and offensive memes datasets. Our experiment results show that DisMultiHate is able to outperform state-of-the-art unimodal and multimodal baselines in the hateful meme classification task. Empirical case studies were also conducted to demonstrate DisMultiHate's ability to disentangle target entities in memes and ultimately showcase DisMultiHate's explainability of the multimodal hateful content classification task.

Finding influential users in online social networks (OSNs) is an important problem with many possible useful applications. Many methods have been proposed to identify influential users in OSNs. PageRank and HITs are two well known examples that determine influential users through link analysis. In recent years, new models that consider both content and social network links have been developed. The Hub and Authority Topic (HAT) model is one that extends HITS to identify topic-specific hubs and authorities by jointly learning hubs, authorities, and topical interests from users' relationship and textual content. However, many of the previous works are confined to identifying influential users within a single OSN. These models, when applied to multiple OSNs, could not learn influential users under a common set of topics nor address platform preferences. In this paper, we therefore propose the MPHAT model, an extension of HAT, to jointly model the topic-specific hub users, authority users, their topical interests and platform preferences. We evaluate MPHAT against several existing state-of-the-art methods in three tasks: (i) modeling of topics, (ii) platform choice prediction, and (iii) link recommendation. Based on our extensive experiments in multiple OSNs settings using synthetic datasets and real-world datasets from Twitter and Instagram, we show that MPHAT is comparable to state-of-the-art topic models in learning topics but outperforms the state-of-the-art models in platform prediction and link recommendation tasks. We also empirically demonstrate the ability of MPHAT to determine influential users within and across multiple OSNs.

While the recent tree-based neural models have demonstrated promising results in generating solution expression for the math word problem (MWP), most of these models do not capture the relationships and order information among the quantities well.This results in poor quantity representations and incorrect solution expressions.In this paper, we propose Graph2Tree, a novel deep learning architecture that combines the merits of the graph-based encoder and tree-based decoder to generate better solution expressions.Included in our Graph2Tree framework are two graphs, namely the Quantity Cell Graph and Quantity Comparison Graph, which are designed to address limitations of existing methods by effectively representing the relationships and order information among the quantities in MWPs.We conduct extensive experiments on two available datasets.Our experiment results show that Graph2Tree outperforms the state-of-the-art baselines on two benchmark datasets significantly.We also discuss case studies and empirically examine Graph2Tree's effectiveness in translating the MWP text into solution expressions 1 .

Academia and industry have developed machine learning and natural language processing models to detect online hate speech automatically.However, most of these existing methods adopt a supervised approach that heavily depends on labeled datasets for training.This results in the methods' poor detection performance of the hate speech class as the training datasets are highly imbalanced.In this paper, we propose HateGAN, a deep generative reinforcement learning model, which addresses the challenge of imbalance class by augmenting the dataset with hateful tweets.We conduct extensive experiments to augment two commonly-used hate speech detection datasets with the HateGAN generated tweets.Our experiment results show that HateGAN improves the detection performance of the hate speech class regardless of the classifiers and datasets used in the detection task.Specifically, we observe an average 5% improvement for the hate class F1 scores across all state-of-the-art hate speech classifiers.We also conduct case studies to empirically examine the HateGAN generated hate speeches and show that the generated tweets are diverse, coherent, and relevant to hate speech detection.

Sketch-Based Image Retrieval (SBIR) is an emerging research area with many real-world applications. Recent studies have approached this research task under the more challenging zero-shot learning setting (ZS-SBIR), which assume classes in the target domain are unseen during the training stage. Many of the existing ZS-SBIR studies transferred the learned cross-modal (i.e., sketch and image) representations from the source domain to the target domain by leveraging side information in semantic embeddings. However, these ZS-SBIR methods are not able to generalize well to a more realistic setting to retrieve images from seen and unseen classes. To address the limitation of existing methods, we propose the cOmmon Conditional Encoder Adversarial Network (OCEAN) to perform generalized zero-shot sketch-based image retrieval (GZS-SBIR). The OCEAN model utilizes a dual learning framework to cyclically map the sketch and image features to a common semantic space, and project semantic features back to the relevant visual space by adversarial training. We conduct experiments on two publicly available datasets and demonstrate that our proposed model outperformed the state-of-the-arts baselines in both ZS-SBIR and GZS-SBIR tasks.

Math word problem (MWP) is challenging due to the limitation in training data where only one “standard” solution is available. MWP models often simply fit this solution rather than truly understand or solve the problem. The generalization of models (to diverse word scenarios) is thus limited. To address this problem, this paper proposes a novel approach, TSN-MD, by leveraging the teacher network to integrate the knowledge of equivalent solution expressions and then to regularize the learning behavior of the student network. In addition, we introduce the multiple-decoder student network to generate multiple candidate solution expressions by which the final answer is voted. In experiments, we conduct extensive comparisons and ablative studies on two large-scale MWP benchmarks, and show that using TSN-MD can surpass the state-of-the-art works by a large margin. More intriguingly, the visualization results demonstrate that TSN-MD not only produces correct final answers but also generates diverse equivalent expressions of the solution.

Social collaborative platforms such as GitHub and Stack Overflow have been increasingly used to improve work productivity via collaborative efforts. To improve user experiences in these platforms, it is desirable to have a recommender system that can suggest not only items (e.g., a GitHub repository) to a user, but also activities to be performed on the suggested items (e.g., forking a repository). To this end, we propose a new approach dubbed Keen2Act, which decomposes the recommendation problem into two stages: the Keen and Act steps. The Keen step identifies, for a given user, a (sub)set of items in which he/she is likely to be interested. The Act step then recommends to the user which activities to perform on the identified set of items. This decomposition provides a practical approach to tackling complex activity recommendation tasks while producing higher recommendation quality. We evaluate our proposed approach using two real-world datasets and obtain promising results whereby Keen2Act outperforms several baseline models.

Increasingly, users are adopting community question-and-answer (Q&A) sites to exchange information. Detecting and eliminating toxic and divisive content in these Q&A sites are paramount tasks to ensure a safe and constructive environment for the users. Insincere question, which is founded upon false premises, is one type of toxic content in Q&A sites. In this paper, we proposed a novel deep learning framework enhanced pre-trained word embeddings with topical information for insincere question classification. We evaluated our proposed framework on a large real-world dataset from Quora Q&A site and showed that the topically enhanced word embedding is able to achieve better results in toxic content classification. An empirical study was also conducted to analyze the topics of the insincere questions on Quora, and we found that topics on "religion", "gender" and "politics" has a higher proportion of insincere questions.

User identity linkage (UIL), the problem of matching user account across multiple online social networks (OSNs), is widely studied and important to many real-world applications. Most existing UIL solutions adopt a supervised or semi-supervised approach which generally suffer from scarcity of labeled data. In this paper, we propose Factoid Embedding, a novel framework that adopts an unsupervised approach. It is designed to cope with different profile attributes, content types and network links of different OSNs. The key idea is that each piece of information about a user identity describes the real identity owner, and thus distinguishes the owner from other users. We represent such a piece of information by a factoid and model it as a triplet consisting of user identity, predicate, and an object or another user identity. By embedding these factoids, we learn the user identity latent representations and link two user identities from different OSNs if they are close to each other in the user embedding space. Our Factoid Embedding algorithm is designed such that as we learn the embedding space, each embedded factoid is "translated" into a motion in the user embedding space to bring similar user identities closer, and different user identities further apart. Extensive experiments are conducted to evaluate Factoid Embedding on two real-world OSNs data sets. The experiment results show that Factoid Embedding outperforms the state-of-the-art methods even without training data.

Finding influential users in online social networks is an important problem with many possible useful applications. HITS and other link analysis methods, in particular, have been often used to identify hub and authority users in web graphs and online social networks. These works, however, have not considered topical aspect of links in their analysis. A straightforward approach to overcome this limitation is to first apply topic models to learn the user topics before applying the HITS algorithm. In this paper, we instead propose a novel topic model known as Hub and Authority Topic (HAT) model to combines the two process so as to jointly learn the hub, authority and topical interests. We evaluate HAT against several existing state-of-the-art methods in two aspects: (i) modeling of topics, and (ii) link recommendation. We conduct experiments on two real-world datasets from Twitter and Instagram. Our experiment results show that HAT is comparable to state-of-the-art topic models in learning topics and it outperforms the state-of-the-art in link recommendation task.

Topic modeling has traditionally been studied for single text collections and applied to social media data represented in the form of text documents. With the emergence of many social media platforms, users find themselves using different social media for posting content and for social interaction. While many topics may be shared across social media platforms, users typically show preferences of certain social media platform(s) over others for certain topics. Such platform preferences may even be found at the individual level. To model social media topics as well as platform preferences of users, we propose a new topic model known as MultiPlatform-LDA (MultiLDA). Instead of just merging all posts from different social media platforms into a single text collection, MultiLDA keeps one text collection for each social media platform but allowing these platforms to share a common set of topics. MultiLDA further learns the user-specific platform preferences for each topic. We evaluate MultiLDA against TwitterLDA, the state-of-the-art method for social media content modeling, on two aspects: (i) the effectiveness in modeling topics across social media platforms, and (ii) the ability to predict platform choices for each post. We conduct experiments on three real-world datasets from Twitter, Instagram and Tumblr sharing a set of common users. Our experiments results show that the MultiLDA outperforms in both topic modeling and platform choice prediction tasks. We also show empirically that among the three social media platforms, "Daily matters" and "Relationship matters" are dominant topics in Twitter, "Social gathering", "Outing" and "Fashion" are dominant topics in Instagram, and "Music", "Entertainment" and "Fashion" are dominant topics in Tumblr.

In an urban city, its transportation network supports efficient flow of people between different parts of the city. Failures in the network can cause major disruptions to commuter and business activities which can result in both significant economic and time losses. In this paper, we investigate the use of centrality measures to determine critical nodes in a transportation network so as to improve the design of the network as well as to devise plans for coping with network failures. Most centrality measures in social network analysis research unfortunately consider only topological structure of the network and are oblivious of transportation factors. This paper proposes a new centrality measure called DelayFlow that incorporates travel time delay and commuter flow volume. We apply the proposed measures on the Singapore's subway network and its about 2 million commuter trips per day, and compare them with traditional topology based centrality measures.