Understanding and inferring causal relationships from texts is a core aspect of human cognition and is essential for advancing large language models (LLMs) towards artificial general intelligence.Existing work evaluating LLM causal reasoning primarily relies on synthetic or simplified texts with explicitly stated causal relationships.These texts typically feature short passages and few causal relations, failing to reflect the complexities of real-world reasoning.In this paper, we investigate whether LLMs are capable of inferring causal relationships from real-world texts.We develop a benchmark drawn from real-world academic literature, which includes diverse texts with respect to length, complexity (different levels of explicitness, number of causal events and relationships), and domain.To the best of our knowledge, our benchmark is the firstever real-world dataset for this task.Our experiments on this dataset show that LLMs face significant challenges in inferring causal relationships from real-world text, with the best-performing model achieving an average F 1 score of only 0.535.Through systematic analysis across aspects of real-world text (explicitness, number of causal events and relationships, length of text, domain), our benchmark offers targeted insights for further research into advancing LLM causal reasoning.

With the development and proliferation of large, complex, black-box models for solving many natural language processing (NLP) tasks, there is also an increasing necessity of methods to stress-test these models and provide some degree of interpretability or explainability. While counterfactual examples are useful in this regard, automated generation of counterfactuals is a data and resource intensive process, that may be infeasible in practice, especially for new tasks and data domains. Therefore, in this work we explore the possibility of leveraging large language models (LLMs) for zero-shot counterfactual generation in order to stress-test NLP models. We propose a structured pipeline to facilitate this generation, and we hypothesize that the instruction-following and textual understanding capabilities of recent LLMs can be effectively leveraged for generating high quality counterfactuals in a zero-shot manner, without requiring any training or fine-tuning. Through comprehensive experiments on a variety of propreitary and open-source LLMs, along with various downstream tasks in NLP, we explore the efficacy of LLMs as zero-shot counterfactual generators in evaluating and explaining black-box NLP models.1

Content moderation on social media faces increasing challenges due to the rapid evolution of hate speech. Identifying hate speech is challenging, especially as it constantly evolves to evade detection. To address this, current methods often rely on auxiliary data like target labels, which specify the particular group targeted by hate speech, to improve detection accuracy. While these target labels can enhance model performance, they are often scarce, inconsistent across platforms, and unable to capture the full spectrum of hate speech variations. To overcome these limitations, we introduce HATE-WATCH, a novel weakly supervised framework that adapts to the fluid nature of hate speech without relying heavily on explicit target labels. By employing confidence-based reweighting and contrastive regularization, HATE-WATCH effectively disentangles input features into universal and platform-specific representations, enabling robust detection even in the absence of detailed target labels. This approach significantly advances cross-platform hate speech detection, offering a more adaptable and scalable solution that contributes to safer online communities by addressing the real-world complexities of content moderation.

Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases, improving their performance in applications like fact-checking and information searching.In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases by injecting deceptive content into the retrieval database, intentionally changing the model's behavior.This threat is critical as it mirrors real-world usage scenarios where RAG systems interact with publicly accessible knowledge bases, such as web scrapings and user-contributed data pools.To be more realistic, we target a realistic setting where the adversary has no knowledge of users' queries, knowledge base data, and the LLM parameters.We demonstrate that it is possible to exploit the model successfully through crafted content uploads with access to the retriever.Our findings emphasize an urgent need for security measures in the design and deployment of RAG systems to prevent potential manipulation and ensure the integrity of machinegenerated content.

espite their value in promoting open discourse, social media plat- forms are often exploited to spread harmful content. Current deep learning and natural language processing models used for detect- ing this harmful content rely on domain-specific terms affecting their ability to adapt to generalizable hate speech detection. This is because they tend to focus too narrowly on particular linguistic signals or the use of certain categories of words. Another signifi- cant challenge arises when platforms lack high-quality annotated data for training, leading to a need for cross-platform models that can adapt to different distribution shifts. Our research introduces a cross-platform hate speech detection model capable of being trained on one platform's data and generalizing to multiple unseen platforms. One way to achieve good generalizability across plat- forms is to disentangle the input representations into invariant and platform-dependent features. We also argue that learning causal relationships, which remain constant across diverse environments, can significantly aid in understanding invariant representations in hate speech. By disentangling input into platform-dependent fea- tures (useful for predicting hate targets) and platform-independent features (used to predict the presence of hate), we learn invariant representations resistant to distribution shifts. These features are then used to predict hate speech across unseen platforms. Our ex- tensive experiments across four platforms highlight our model's enhanced efficacy compared to existing state-of-the-art methods in detecting generalized hate speech

Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, Huan Liu. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

The evergrowing reliance of humans and society on machine learning methods has raised concerns about their trustworthiness and liability. As a response to these concerns, Socially Responsible Machine Learning (SRML) aims at developing fair, transparent, and robust machine learning algorithms. However, traditional approaches to SRML do not incorporate human perspectives, and therefore are not sufficient to build long-lasting trust between machines and human being. Causality as the key to human intelligence plays a vital role in achieving socially responsible machine learning algorithms which are compatible with human notions. Bridging the gap between traditional SRML and causality, in this tutorial, we aim at providing a holistic overview of SRML through the lens of causality. In particular, we will focus on state-of-the-art techniques on causal socially responsible ML in terms of fairness, interpretability, and robustness. The objectives of this tutorial are as follows: (1) we provide a taxonomy of existing literature on causal socially responsible ML from fairness, interpretability, and robustness perspective; (2) we review the state-of-the-art techniques for each task; and (3) we elucidate open questions and future research directions. We believe this tutorial is beneficial to researchers and practitioners from the areas of data mining, machine learning, and social sciences.

Deep Neural Networks are shown to be prone to adversarial attacks. In the black-box setting, where no information about the target is available, surrogate-based black-box attacks train a surrogate on samples queried from the target to imitate the black-box’s behavior. The trained surrogate is then attacked to generate adversarial examples. Existing surrogate-based attacks suffer from low success rates because they fail to accurately capture the target’s behavior, i.e., their surrogates only mimic the target’s outputs for a given set of inputs. Moreover, their attack strategy relies on noisy estimations of high dimensional gradients w.r.t. the inputs (i.e., surrogate’s gradients) to generate adversarial examples. Ideally, a successful surrogate-based attack should possess two properties: (1) Train and employ a surrogate that accurately imitates the target behavior for every pair of input and output, i.e., the joint distribution of the target over its input and outputs; and (2) Generate adversarial examples by directly manipulating the class-dependent factors of the input, i.e., factors that affect the target’s output, rather than relying on noisy estimations of gradients. We propose a novel surrogate-based attack framework with a surrogate architecture that learns the target distribution over its inputs and outputs while disentangling the class-dependent factors from class-irrelevant ones. The framework is equipped with a novel attack strategy that fully utilizes the target distribution captured by the surrogate while generating adversarial examples by directly manipulating the class-dependent factors. Extensive experiments demonstrate the efficacy of our attack in generating highly successful adversarial examples compared to state-of-the-art methods.

Adversarial attacks have recently been proposed to scrutinize the security of deep neural networks. Most blackbox adversarial attacks, which have partial access to the target through queries, are target-specific; e.g., they require a well-trained surrogate that accurately mimics a given target. In contrast, target-agnostic black-box attacks are developed to attack any target; e.g., they learn a generalized surrogate that can adapt to any target via fine-tuning on samples queried from the target. Despite their success, current state-of-the-art target-agnostic attacks require tremendous fine-tuning steps and consequently an immense number of queries to the target to generate successful attacks. The high query complexity of these attacks makes them easily detectable and thus defendable. We propose a novel query-efficient target-agnostic attack that trains a generalized surrogate network to output the adversarial directions iv.r.t. the inputs and equip it with an effective fine-tuning strategy that only fine-tunes the surrogate when it fails to provide useful directions to generate the attacks. Particularly, we show that to effectively adapt to any target and generate successful attacks, it is sufficient to fine-tune the surrogate with informative samples that help the surrogate get out of the failure mode with additional information on the target’s local behavior. Extensive experiments on CIFAR10 and CIFAR-100 datasets demonstrate that the proposed target-agnostic approach can generate highly successful attacks for any target network with very few fine-tuning steps and thus significantly smaller number of queries (reduced by several order of magnitudes) compared to the state-of-the-art baselines.

Modeling spillover effects from observational data is an important problem in economics, business, and other fields of research. It helps us infer the causality between two seemingly unrelated set of events. For example, if consumer spending in the United States declines, it has spillover effects on economies that depend on the U.S. as their largest export market. In this paper, we aim to infer the causation that results in spillover effects between pairs of entities (or units); we call this effect as paired spillover. To achieve this, we leverage the recent developments in variational inference and deep learning techniques to propose a generative model called Linked Causal Variational Autoencoder (LCVA). Similar to variational autoencoders (VAE), LCVA incorporates an encoder neural network to learn the latent attributes and a decoder network to reconstruct the inputs. However, unlike VAE, LCVA treats the latent attributes as confounders that are assumed to affect both the treatment and the outcome of units. Specifically, given a pair of units u and $\baru $, their individual treatment and outcomes, the encoder network of LCVA samples the confounders by conditioning on the observed covariates of u, the treatments of both u and $\baru $ and the outcome of u. Once inferred, the latent attributes (or confounders) of u captures the spillover effect of $\baru $ on u. Using a network of users from job training dataset (LaLonde (1986)) and co-purchase dataset from Amazon e-commerce domain, we show that LCVA is significantly more robust than existing methods in capturing spillover effects.

Personalized recommendation has become popular in modern web services. For instance, Amazon recommends new items to shoppers. Also, Netflix recommends shows to viewers, and Facebook recommends friends to its users. Despite the ubiquity of recommendation applications, classic database management systems still do not provide in-house support for recommending data stored in the database. In this paper, we present the anatomy of RecDB an open source PostgreSQLbased system that provides a unified approach for declarative data recommendation inside the database engine. RecDB realizes the personalized recommendation functionality as query operators inside the database kernel. That facilitates applying the recommendation functionality and typical database operations (e.g., Selection, Join, Top-k) side-by-side. To further reduce the application latency, RecDB pre-computes and caches the generated recommendation in the database. In the paper, we present extensive experiments that study the performance of personalized recommendation applications based on an actual implementation inside PostgreSQL 9.2 using real Movie recommendation and location-aware recommendation scenarios. The results show that a recommendation-aware database engine, i.e., RecDB, outperforms the classic approach that implements the recommendation logic on-top of the database engine in various recommendation applications.

The paper demonstrates Hippo a lightweight database indexing scheme that significantly reduces the storage and maintenance overhead without compromising much on the query execution performance. Hippo stores disk page ranges instead of tuple pointers in the indexed table to reduce the storage space occupied by the index. It maintains simplified histograms that represent the data distribution and adopts a page grouping technique that groups contiguous pages into page ranges based on the similarity of their index key attribute distributions. When a query is issued, Hippo leverages the page ranges and histogram-based page summaries to recognize those pages such that their tuples are guaranteed not to satisfy the query predicates and then inspects the remaining pages. We demonstrate Hippo using a billion NYC taxi trip records. Video: http://www.youtube.com/watch?v=wWaOK2-9k9A.