Large language models struggle with reasoning over long contexts where relevant information is sparsely distributed.Although planand-execute frameworks mitigate this by decomposing tasks into planning and execution, their effectiveness is often limited by unreliable plan generation due to dependence on surfacelevel cues.Consequently, plans may be based on incorrect assumptions, and once a plan is formed, identifying errors and revising it reliably becomes difficult, limiting the effectiveness of reactive refinement.To address this limitation, we propose PPA-Plan, a proactive planning strategy for long-context reasoning that focuses on preventing such failures before plan generation.PPA-Plan identifies potential logical pitfalls and false assumptions, formulates them as negative constraints, and conditions plan generation on explicitly avoiding these constraints.Experiments on long-context QA benchmarks show that executing plans generated by PPA-Plan consistently outperforms existing plan-and-execute methods and direct prompting.Query: How many times did Burnett operate the claw in the passage?1. burnett_claw_operations = FIND_ELEMENT(CTX, "operate", "Burnett claw") 2. count_operations = COUNT_X(CTX, burnett_claw_operations) Answer: He directs the claw to retrieve Kriere from space.Then, he operates the claw to close around Kriere and crush him.So, the number of times Burnett operates the claw is 2. Query: Why was the murderer trying to kill Bo?⚠ Negative Constraints: -Assuming the motive for the murder is explicitly stated, rather than needing to infer it from context clues or character interactions.-Assuming that the murderer has a single, clear motive, overlooking the possibility of multiple, complex reasons for their actions.Plan ① FIND_RELATION ② FIND_ELEMENT ③ FIND_EVENT ④ CONCAT 1. murderer_relationship = FIND_RELATION(CTX,"murderer", "Bo") 2. motive_clues = FIND_ELEMENT(CTX, "motive clues", "murderer") 3. past_events = FIND_EVENT(CTX, "murderer") 4. combined_motive = CONCAT(murderer_relationship, motive_clues, past_events) Answer: …(details omitted).While the text does not provide explicit details about the murderer's motives, it suggests a broader context of violence and rivalry among individuals in the asteroid belt.Example (a) Planner Plan ① FIND_ELEMENT ② COUNT_X Executor Result 1 Result 2Query: Why was the murderer trying to kill Bo? 1. murderer_intent = FIND_INTENT(CTX, "murderer") 2. bo_relationship = FIND_RELATION(CTX, "Bo", "murderer") 3. motive = FIND_ELEMENT(CTX, "motive", "murderer") 4. ans = CONCAT(murderer_intent, bo_relationship, motive) Planner Executor

Listwise reranking with large language models (LLMs) enhances top-ranked results in retrieval-based applications. Due to the limit in context size and high inference cost of long context, reranking is typically performed over a fixed size of small subsets, with the final ranking aggregated from these partial results. This fixed computation disregards query difficulty and document distribution, leading to inefficiencies. We propose AcuRank, an adaptive reranking framework that dynamically adjusts both the amount and target of computation based on uncertainty estimates over document relevance. Using a Bayesian TrueSkill model, we iteratively refine relevance estimates until reaching sufficient confidence levels, and our explicit modeling of ranking uncertainty enables principled control over reranking behavior and avoids unnecessary updates to confident predictions. Results on the TREC-DL and BEIR benchmarks show that our method consistently achieves a superior accuracy–efficiency trade-off and scales better with compute than fixed-computation baselines. These results highlight the effectiveness and generalizability of our method across diverse retrieval tasks and LLM-based reranking models.

Soyoung Yoon, Sungjoon Park, Gyuwan Kim, Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, Alice Oh. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar.Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model.Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model.In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions.This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary.Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark 1 .

Gyuwan Kim, Kyunghyun Cho. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-level text as an input, ST-BERT learns a contextualized cross-modal alignment via our two proposed pre-training tasks: Cross-modal Masked Language Modeling (CM-MLM) and Cross-modal Conditioned Language Modeling (CM-CLM). Experimental results on three benchmarks present that our approach is effective for various SLU datasets and shows a surprisingly marginal performance degradation even when 1% of the training data are available. Also, our method shows further SLU performance gain via domain-adaptive pre-training with domain-specific speech-text pair data.

End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or finetuning with knowledge distillation. To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterancelevel representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially. We use vqwav2vec BERT as a speech encoder because it captures general and rich features. Furthermore, we improve the performance, especially in a low-resource scenario, with data augmentation methods by randomly masking spans of discrete audio tokens and contextualized hidden representations. Consequently, we push the state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy in the full dataset setting and 99.5% in the 10% subset setting. Throughout the ablation studies, we empirically verify that all used methods are crucial to the final performance, providing the best practice for spoken language understanding. Code is available at https://github.com/clovaai/textual-kd-slu.

Recent works in dialogue state tracking (DST) focus on an open vocabulary-based setting to resolve scalability and generalization issues of the predefined ontology-based approaches.However, they are inefficient in that they predict the dialogue state at every turn from scratch.Here, we consider dialogue state as an explicit fixed-sized memory and propose a selectively overwriting mechanism for more efficient DST.This mechanism consists of two steps: (1) predicting state operation on each of the memory slots, and (2) overwriting the memory with new values, of which only a few are generated according to the predicted state operations.Our method decomposes DST into two sub-tasks and guides the decoder to focus only on one of the tasks, thus reducing the burden of the decoder.This enhances the effectiveness of training and DST performance.Our SOM-DST (Selectively Overwriting Memory for Dialogue State Tracking) model achieves state-of-theart joint goal accuracy with 51.72% in Mul-tiWOZ 2.0 and 53.01% in MultiWOZ 2.1 in an open vocabulary-based DST setting.In addition, we analyze the accuracy gaps between the current and the ground truth-given situations and suggest that it is a promising direction to improve state operation prediction to boost the DST performance. 1

Current neural query auto-completion (QAC) systems rely on character-level language models, but they slow down when queries are long.We present how to utilize subword language models for the fast and accurate generation of query completion candidates.Representing queries with subwords shorten a decoding length significantly.To deal with issues coming from introducing subword language model, we develop a retrace algorithm and a reranking method by approximate marginalization.As a result, our model achieves up to 2.5 times faster while maintaining a similar quality of generated results compared to the character-level baseline.Also, we propose a new evaluation metric, mean recoverable length (MRL), measuring how many upcoming characters the model could complete correctly.It provides more explicit meaning and eliminates the need for prefix length sampling for existing rank-based metrics.Moreover, we performed a comprehensive analysis with ablation study to figure out the importance of each component 1 .