The quality of rationales is essential in the reasoning capabilities of language models. Rationales not only enhance reasoning performance in complex natural language tasks but also justify model decisions. However, obtaining impeccable rationales is often impossible. Our study aims to investigate how ambiguous rationales play in model performances of natural language reasoning. We first assess the ambiguity of rationales through the lens of entropy and uncertainty in model prior beliefs, exploring its impact on task performance. We then propose a simple way to guide models to choose between two different reasoning paths depending on the ambiguity of the rationales. Our empirical results demonstrate that this approach leads to robust performance, particularly in adversarial scenarios where rationale quality is inconsistent.

Large language models (LLMs) frequently generate confident yet inaccurate responses, introducing significant risks for deployment in safety-critical domains.We present a novel, test-time approach to detecting model hallucination through systematic analysis of information flow across model layers.We target cases when LLMs process inputs with ambiguous or insufficient context.Our investigation reveals that hallucination manifests as usable information deficiencies in inter-layer transmissions.While existing approaches primarily focus on final-layer output analysis, we demonstrate that tracking cross-layer information dynamics (LI) provides robust indicators of model reliability, accounting for both information gain and loss during computation.LI integrates easily with pretrained LLMs without requiring additional training or architectural modifications.

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as safety' androbustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.

Solving math word problems depends on how to articulate the problems, the lens through which models view human linguistic expressions.Real-world settings count on such a method even more due to the diverse practices of the same mathematical operations.Earlier works constrain available thinking processes by limited prediction strategies without considering their significance in acquiring mathematical knowledge.We introduce Attention-based THought Expansion Network Architecture (ATHENA) to tackle the challenges of real-world practices by mimicking human thought expansion mechanisms in the form of neural network propagation.A thought expansion recurrently generates the candidates carrying the thoughts of possible math expressions driven from the previous step and yields reasonable thoughts by selecting the valid pathways to the goal.Our experiments show that ATHENA achieves a new state-of-the-art stage toward the ideal model that is compelling in variant questions even when the informativeness in training examples is restricted. 1 Context The school playground was originally [80] meters long and [40] meters wide.Later when the school is remodeled, the length is increased by [10] meters and the width is increased by [15] meters.Train on an example of a question-solution pair under the context above.Question How many square meters is the original playground area?Solution (80 × 40)Test on variant questions that share the context above.Q0 How many times the length of the original playground was the width?DeductReasoner ATHENA UnbiasedMWP (80 + 10) × (40 + 15) -(80 × 40) (X) UnbiasedMWP (80 + 10) × (40 -15) (X) UnbiasedMWP (1:N) 80 ÷ 40 (O) UnbiasedMWP (1:N) 80 ÷ 40 (O) Q1 How many square meters is the current playground area?DeductReasoner ATHENA UnbiasedMWP (80 + 10) × (40 + 15) -(80 × 40) (X) UnbiasedMWP (80 + 10) × (40 + 15) (O) UnbiasedMWP (1:N) 80 × 40 (X) UnbiasedMWP (1:N) (80 + 10) × (40 + 15) (O) Q2 How many square meters are increased by the current playground area compared to the original one?DeductReasoner ATHENA UnbiasedMWP (80 + 10) × (40 + 15) -(80 × 40) (O) UnbiasedMWP (80 + 10) × (40 + 15) -(80 × 40) (O) UnbiasedMWP (1:N) 80 × 40 (X) UnbiasedMWP (1:N) (80 + 10) × (40 + 15) -(80 × 40) (O) An example with a lexically similar context to that of above from the UnbiasedMWP Context The school basketball court was [20] meters long and [12] meters wide.After the renovation, the length is increased by [8] meters, and the width increases by [3] meters.Question How many square meters are increased?Solution (20 + 8) × (12 + 3) -(20 × 12)

Data augmentation has been an important ingredient for boosting performances of learned models. Prior data augmentation methods for few-shot text classification have led to great performance boosts. However, they have not been designed to capture the intricate compositional structure of natural language. As a result, they fail to generate samples with plausible and diverse sentence structures. Motivated by this, we present the data Augmentation using Lexicalized Probabilistic context-free grammars (ALP) that generates augmented samples with diverse syntactic structures with plausible grammar. The lexicalized PCFG parse trees consider both the constituents and dependencies to produce a syntactic frame that maximizes a variety of word choices in a syntactically preservable manner without specific domain experts. Experiments on few-shot text classification tasks demonstrate that ALP enhances many state-of-the-art classification methods. As a second contribution, we delve into the train-val splitting methodologies when a data augmentation method comes into play. We argue empirically that the traditional splitting of training and validation sets is sub-optimal compared to our novel augmentation-based splitting strategies that further expand the training split with the same number of labeled data. Taken together, our contributions on the data augmentation strategies yield a strong training recipe for few-shot text classification tasks.