Large language models (LLMs) have achieved impressive progress in natural language processing tasks but still struggle with complex logical reasoning. We observe that in propositional logic question-answering (QA), LLMs' performance varies with the order of training samples during fine-tuning. Motivated by this, we propose a data-driven approach to automatically determine the fine-tuning sample order, enhancing the logical QA performance of LLMs. Specifically, we first quantify the logical reasoning complexity of propositional reasoning samples and then stratify the training data into several subsets of ascending complexity. Subsequently, we fine-tune the LLMs on these subsets, progressing from low to high reasoning complexity. Experimental results demonstrate that our approach outperforms single-stage fine-tuning baselines across diverse reasoning benchmarks.

Large language models (LLMs) have achieved remarkable successes on various tasks. However, recent studies have found that there are still significant challenges to the logical reasoning abilities of LLMs, which can be categorized into the following two aspects: (1) Logical question answering: LLMs often fail to generate the correct answer within a complex logical problem which requires sophisticated deductive, inductive or abductive reasoning given a collection of premises and constrains. (2) Logical consistency: LLMs are prone to producing responses contradicting themselves across different questions. For example, a state-of-the-art question-answering LLM Macaw, answers Yes to both questions Is a magpie a bird? and Does a bird have wings? but answers No to Does a magpie have wings?. To facilitate this research direction, we comprehensively investigate the most cutting-edge methods and propose a detailed taxonomy. Specifically, to accurately answer complex logic questions, previous methods can be categorized based on reliance on external solvers, prompts, and fine-tuning. To avoid logical contradictions, we discuss concepts and solutions of various logical consistencies, including implication, negation, transitivity, factuality consistencies, and their composites. In addition, we review commonly used benchmark datasets and evaluation metrics, and discuss promising research directions, such as extending to modal logic to account for uncertainty and developing efficient algorithms that simultaneously satisfy multiple logical consistencies.

Warning: this paper contains content that may be offensive or upsetting.In this paper, we investigate what types of stereotypical information are captured by pretrained language models.We present the first dataset comprising stereotypical attributes of a range of social groups and propose a method to elicit stereotypes encoded by pretrained language models in an unsupervised fashion.Moreover, we link the emergent stereotypes to their manifestation as basic emotions as a means to study their emotional effects in a more generalized manner.To demonstrate how our methods can be used to analyze emotion and stereotype shifts due to linguistic experience, we use fine-tuning on news sources as a case study.Our experiments expose how attitudes towards different social groups vary across models and how quickly emotions and stereotypes can shift at the fine-tuning stage.

Linguistic pragmatics assumes that conversation is a by-and-large cooperative endeavour. Although clearly reasonable and helpful, this is an idealization and it pays to ask what happens to natural language interpretation if the presumption of cooperativity is dropped, be that entirely or only to some degree. Game theory suggests itself as a formal tool for modelling the different degrees in which speaker and hearer may or may not have common interests, and it is in this game-theoretic light that this article investigates in particular a notion of speaker-relevance and its impact on the question why we communicate cooperatively in most cases and what happens to pragmatic phenomena such as conversational implicatures if full cooperation cannot be assumed.

We give derivations of two formal models of Gricean Quantity1 implicature and strong exhaustivity (Van Rooij and Schulz, 2004; Schulz and Van Rooij, 2006), in bidirectional optimality theory and in a signalling games framework. We show that, under a unifying model based on signalling games, these interpretative strategies are game-theoretic equilibria when the speaker is known to be respectively minimally and maximally expert in the matter at hand. That is, in this framework the optimal strategy for communication depends on the degree of knowledge the speaker is known to have concerning the question she is answering.In addition, and most importantly, we give a game-theoretic characterisation of the interpretation rule Grice (formalising Quantity1 implicature), showing that under natural conditions this interpretation rule occurs in the unique equilibrium play of the signalling game.