Large Language Models (LLMs) require robust evaluation.However, existing frameworks often rely on curated datasets that, once public, may be accessed by newer LLMs.This creates a risk of data leakage, where test sets inadvertently become part of training data, compromising evaluation fairness and integrity.To mitigate this issue, we propose Behave as Claimed (BaC), a novel evaluation framework inspired by counterfactual reasoning.BaC constructs a "what-if" scenario where LLMs respond to counterfactual questions about how they would behave if the input were manipulated.We refer to these responses as claims, which are verifiable by observing the LLMs' actual behavior when given the manipulated input.BaC dynamically generates and verifies counterfactual questions using various few-shot in-context learning evaluation datasets, reducing their susceptibility to data leakage.Moreover, BaC provides a more challenging evaluation paradigm for LLMs.LLMs must thoroughly understand the prompt, the task, and the consequences of their responses to achieve better performance.We evaluate several LLMs and find that, while most perform well on the original datasets, they struggle with BaC.This suggests that LLMs usually fail to align their claims with their actual behavior and that high performance on standard datasets may be less stable than previously assumed.

Stacked AutoEncoders (SAE) have been widely adopted in edge anomaly detection scenarios. However, the resource-intensive nature of SAE can pose significant challenges for edge devices, which are typically resource-constrained and must adapt rapidly to dynamic and changing conditions. Optimizing SAE to meet the heterogeneous demands of real-world deployment scenarios, including high performance under constrained storage, low power consumption, fast inference, and efficient model updates, remains a substantial challenge. To address this, we propose an integrated optimization framework that jointly considers these critical factors to achieve balanced and adaptive system-level optimization. Specifically, we formulate SAE optimization for edge anomaly detection as a multi-objective optimization problem and propose MO-SAE (Multi-Objective Stacked AutoEncoders). The multiple objectives are addressed by integrating model clipping, multi-branch exit design, and a matrix approximation technique. In addition, a multi-objective heuristic algorithm is employed to effectively balance the competing objectives in SAE optimization. Our results demonstrate that the proposed MO-SAE delivers substantial improvements over the original approach. On the x86 architecture, it reduces storage space and power consumption by at least 50%, improves runtime efficiency by no less than 28%, and achieves an 11.8% compression rate, all while maintaining application performance. Furthermore, MO-SAE runs efficiently on edge devices with ARM architecture. Experimental results show a 15% improvement in inference speed, facilitating efficient deployment in cloud–edge collaborative anomaly detection systems.

Multi-hop questions are associated with a series of justifications, and one needs to obtain the answers by following the reasoning path (RP) that orders the justifications adequately. So reasoning path retrieval becomes a critical preliminary stage for multi-hop Question Answering (QA). Within the RP, two fundamental challenges emerge for better performance: (i) what the order of the justifications in the RP should be, and (ii) what if the wrong justification has been in the path. In this paper, we propose Reasoning Path Augmentation (RPA), which uses reasoning path reordering and augmentation to handle the above two challenges, respectively. Reasoning path reordering restructures the reasoning by targeting the easier justification first but difficult one later, in which the difficulty is determined by the overlap between query and justifications since the higher overlap means more lexical relevance and easier searchable. Reasoning path augmentation automatically generates artificial RPs, in which the distracted justifications are inserted to aid the model recover from the wrong justification. We build RPA with a naive pre-trained model and evaluate RPA on the QASC and MultiRC datasets. The evaluation results demonstrate that RPA outperforms previously published reasoning path retrieval methods, showing the effectiveness of the proposed methods. Moreover, we present detailed experiments on how the orders of justifications and the percent of augmented paths affect the question- answering performance, revealing the importance of polishing RPs and the necessity of augmentation.

In recent years, neural networks have achieved impressive performance on dialogue response generation. However, most of these models still suffer from some shortcomings, such as yielding uninformative responses and lacking explainable ability. This article proposes a Two-stage Dialogue Response Generation model (TSRG), which specifies a method to generate diverse and informative responses based on an interpretable procedure between stages. TSRG involves a two-stage framework that generates a candidate response first and then instantiates it as the final response. The positional information and a resident token are injected into the candidate response to stabilize the multi-stage framework, alleviating the shortcomings in the multi-stage framework. Additionally, TSRG allows adjusting and interpreting the interaction pattern between the two generation stages, making the generation response somewhat explainable and controllable. We evaluate the proposed model on three dialogue datasets that contain millions of single-turn message-response pairs between web users. The results show that, compared with the previous multi-stage dialogue generation models, TSRG can produce more diverse and informative responses and maintain fluency and relevance.

Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge.However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge.To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content.The deterministic relationship ensures that the masked factual content can be deterministically inferable based on the existing clues in the context.That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking.Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks.Specifically, we use an external Knowledge Base (KB) to identify deterministic relationships and continuously pre-train PLMs with the proposed methods.The factual knowledge probing experiments indicate that the continuously pre-trained PLMs achieve better robustness in factual knowledge capturing.Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.

Collecting supporting evidence from large corpora of text (e.g., Wikipedia) is of great challenge for open-domain Question Answering (QA). Especially, for multi-hop open-domain QA, scattered evidence pieces are required to be gathered together to support the answer extraction. In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering. Specifically, the hop in this paper is defined as the combination of a hyperlink and the corresponding outbound link document. The hyperlink is encoded as the mention embedding which models the structured knowledge of how the outbound link entity is mentioned in the textual context, and the corresponding outbound link document is encoded as the document embedding representing the unstructured knowledge within it. Accordingly, we build HopRetriever which retrieves hops over Wikipedia to answer complex questions. Experiments on the HotpotQA dataset demonstrate that HopRetriever outperforms previously published evidence retrieval methods by large margins. Moreover, our approach also yields quantifiable interpretations of the evidence collection process.