Single-cell RNA sequencing data enables analysis of transcript levels of single cells across different cell types and conditions. Recent work has highlighted the value of measuring gene-specific transcriptional variability, or noise, within a genetically identical population of cells in addition to mean expression, given that these differences contribute to biological processes including development and disease. However, measuring transcriptional noise remains a challenge. Here, we systematically compared statistical methods by simulating single-cell data by varying both dispersion and count size to assess the relative responsiveness to noise of several commonly used statistical metrics: the Gini index, variance-to-mean ratio, variance, coefficient of variance (CV), CV2, and Shannon entropy. We found that the variance-to-mean ratio scales approximately linearly with increasing dispersion and is independent of dataset size. In contrast, the Gini index displayed paradoxical behavior in that it increases as dispersion decreases, and Shannon entropy was not scale-invariant. Next, we applied the variance-to-mean ratio (Fano factor) to measure transcriptional variability in single-cell datasets representing different complex systems and cross-platform measurements. Our data show that many genes display transcriptional variability within the same cell type, and that while variation does not correlate with gene characteristics such as transcript level, promoter GC content, or evolutionary gene age, variable genes are often correlated with specific biological processes. Notably, most genes and pathways with highest transcriptional variability as identified by the Fano factor were largely independent of differentially expressed genes and have also been implicated in biological processes related to the system. Thus, our data highlight that choice and application of appropriate models for measuring transcriptional variation in scRNA-seq data can reveal biologically relevant information beyond what is observed from mean expression alone.

Malicious content generated by large language models (LLMs) can pose varying degrees of harm. Although existing LLM-based moderators can detect harmful content, they struggle to assess risk levels and may miss lower-risk outputs. Accurate risk assessment allows platforms with different safety thresholds to tailor content filtering and rejection. In this paper, we introduce per-topic severity rubrics for 11 harmful topics and build BingoGuard, an LLM-based moderation system designed to predict both binary safety labels and severity levels. To address the lack of annotations on levels of severity, we propose a scalable generate-then-filter framework that first generates responses across different severity levels and then filters out low-quality responses. Using this framework, we create BingoGuardTrain, a training dataset with 54,897 examples covering a variety of topics, response severity, styles, and BingoGuardTest, a test set with 988 examples explicitly labeled based on our severity rubrics that enables fine-grained analysis on model behaviors on different severity levels. Our BingoGuard-8B, trained on BingoGuardTrain, achieves the state-of-the-art performance on several moderation benchmarks, including WildGuardTest and HarmBench, as well as BingoGuardTest, outperforming best public models, WildGuard, by 4.3\%. Our analysis demonstrates that incorporating severity levels into training significantly enhances detection performance and enables the model to effectively gauge the severity of harmful responses. Warning: this paper includes red-teaming examples that may be harmful in nature.

Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT) [64], which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose Torrance Test of Creative Writing (TTCW) to evaluate creativity as product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.

Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

In an Opportunistic Network (Oppnet), the transmission of messages between mobile devices is achieved in a store-carry-and-forward fashion since nodes store the incoming messages in their buffer and wait until a suitable next hop node is encountered that can carry the message closer to the destination. In such environment, due to the delay-tolerant nature of the network, designing a routing protocol is a challenge. This paper proposes a novel routing protocol called Encounter and Distance based Routing (EDR), which utilizes the so-called forward parameter to determine the next hop selection. This parameter is calculated by taking into account the number of encounters and the distance of each node in the network with respect to a particular destination. Simulation results are provided, showing the superiority of EDR over the History based Prediction for Routing (HBPR) protocol and the ProWait protocol, chosen as benchmark schemes, in terms of hop count, messages dropped, and average latency.