Recent progress in large language models (LLMs) has enabled them to express their confidence in natural language, improving transparency and reliability.However, this expressiveness is often accompanied by systematic overconfidence, whose underlying causes remain poorly understood.In this work, we analyze the dynamics of verbalized confidence estimation and identify answer-independence-the failure to condition confidence on the model's own answer-as a primary driver of this behavior.To address this, we introduce ADVICE (Answer-Dependent VerbalIzed Confidence Estimation), a fine-tuning framework that promotes answer-grounded confidence estimation.Extensive experiments show that ADVICE substantially improves confidence calibration, while exhibiting strong generalization to unseen settings without degrading task performance.We further demonstrate that these gains stem from enhanced answer dependence, shedding light on the origins of overconfidence and enabling trustworthy confidence verbalization.

This study proposes a novel coding framework for analyzing science classroom discourse using large language models (LLMs), adopting fine-grained utterance-level chunking aligned with the analytical units of LLMs to address limitations of global, lesson-level observation tools. Authentic middle school science classroom discourse was annotated through a dual-expertise and iterative process integrating the theoretical knowledge of science education faculty with the experiential insights of in-service teachers, supported by systematic rater training to ensure conceptual alignment and interpretive consistency at the utterance level. Through this process, a science education glossary comprising 137 instructional terms organized into 20 thematic categories was developed using a primarily bottom-up approach informed by established observation frameworks. Building on this theory-informed foundation, we systematically examined LLM-based methods for predicting instructional themes and quality, comparing structured prompting strategies with domain-adaptive fine-tuning across model architectures. These contributions lay a foundation for future research on interpretable, scalable, and pedagogically meaningful automated formative feedback to support teachers’ self-reflection and professional growth.

MOTIVATION: Named entity recognition (NER) is a fundamental component of structured knowledge extraction, yet its effectiveness in emerging domains remains by the scarcity of high-quality, domain-specific annotated corpora. Although data augmentation and distant supervision have been explored to alleviate this issue, existing methods often introduce limited entity diversity, noisy labels, or disrupt contextual integrity, thereby limiting their generalization ability in low-resource settings. RESULTS: In this study, we propose DA-BioNER, a context-preserving data expansion framework for biomedical NER. DA-BioNER combines multiple base NER models trained on few-shot data to provide coarse annotations, followed by refinement using a large language model (LLM) guided by global biomedical knowledge. Unlike generation-based augmentation methods that synthesize new sentences, DA-BioNER performs annotation refinement within existing sentences, preserving both syntactic structure and semantic context. By constraining the role of LLM to refinement rather than open-ended generation, the framework effectively reduces hallucination while improving label precision and consistency. We evaluate DA-BioNER on three benchmark datasets (NCBI-Disease, BC5CDR, and BioRED), under low-resource conditions. In 40-shot settings, DA-BioNER achieves F1-scores of 0.750, 0.795, and 0.799, respectively, outperforming state-of-the-art methods, including LSMS, DAGA, and MELM, by up to 0.32. Under more extreme few-shot settings, DA-BioNER further improves F1-scores by up to 0.08, while generating an average of 1,391 additional unique entities, substantially enriching training diversity. These results demonstrate that DA-BioNER provides a scalable and adaptable solution for robust biomedical NER, particularly in domain adaptation and low-resource scenarios. AVAILABILITY: DA-BioNER is publicly available at https://github.com/DMnBI/DA-BioNER.

Conversational Recommender Systems (CRS) aim to provide tailored recommendation responses via a chat interface, including both the user's preferred item and its accompanying explanation. However, due to its generative nature, CRS are prone to responding with factually incorrect explanations (i.e., hallucinations). To solve this problem, we propose incorporating a passage retrieval module into CRS with the objective of enhancing the factuality and informativeness of system responses. Specifically, we outline essential directions for employing a passage retrieval module in CRS to address the following critical issues: (1) the risk of passage retrieval not aligning with the user preference; (2) the absence of supervision for training a passage retrieval module. As a solution, we introduce ESPRESSO, a novel passage retrieval approach for CRS, to effectively tackle the above issues with two core ideas: adaptive item selection and relevance-based groupwise learning. Our extensive experiments show that ESPRESSO effectively resolves issues, achieving up to 36% higher Hit@3 accuracy than the best of 8 competing methods. Additionally, we verify that leveraging passages retrieved by ESPRESSO significantly improves the response quality of CRS.

Real-world decision-making often requires integrating and reasoning over information from multiple modalities. While recent multimodal large language models (MLLMs) have shown promise in such tasks, their ability to perform multi-hop reasoning across diverse sources remains insufficiently evaluated. Existing benchmarks, such as MMQA, face challenges due to (1) data contamination and (2) a lack of complex queries that necessitate operations across more than two modalities, hindering accurate performance assessment. To address this, we present Financial Cross-Modal Multi-Hop Reasoning (FCMR), a benchmark created to analyze the reasoning capabilities of MLLMs by urging them to combine information from textual reports, tables, and charts within the financial domain. FCMR is categorized into three difficulty levels—Easy, Medium, and Hard—facilitating a step-by-step evaluation. In particular, problems at the Hard level require precise cross-modal three-hop reasoning and are designed to prevent the disregard of any modality. Experiments on this new benchmark reveal that even state-of-the-art MLLMs struggle, with the best-performing model (Claude 3.5 Sonnet) achieving only 30.4% accuracy on the most challenging tier. We also conduct analysis to provide insights into the inner workings of the models, including the discovery of a critical bottleneck in the information retrieval phase.

Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks by leveraging pre-trained (i.e., parametric) and external (i.e., contextual) knowledge. While substantial efforts have been made to enhance the utilization of both forms of knowledge, situations in which models lack relevant information remain underexplored. To investigate this challenge, we first present a controlled testbed featuring four distinct knowledge access scenarios, including the aforementioned edge case, revealing that conventional LLM usage exhibits insufficient robustness in handling all instances. Addressing this limitation, we propose Contrastive Decoding with Abstention (CDA), a novel training-free decoding method that allows LLMs to generate responses when relevant knowledge is available and to abstain otherwise. CDA estimates the relevance of both knowledge sources for a given input, adaptively deciding which type of information to prioritize and which to exclude. Through extensive experiments, we demonstrate that CDA can effectively perform accurate generation and abstention simultaneously, enhancing reliability and preserving user trust.

Idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions.While recent studies have leveraged large language models (LLMs) to handle idioms across various tasks, e.g., idiom-containing sentence generation and idiomatic machine translation, little is known about the underlying mechanisms of idiom processing in LLMs, particularly in multilingual settings.To this end, we introduce MI-DAS, a new large-scale dataset of idioms in six languages, each paired with its corresponding meaning.Leveraging this resource, we conduct a comprehensive evaluation of LLMs' idiom processing ability, identifying key factors that influence their performance.Our findings suggest that LLMs rely not only on memorization but also adopt a hybrid approach that integrates contextual cues and reasoning, especially when processing compositional idioms.This implies that idiom understanding in LLMs emerges from an interplay between internal knowledge retrieval and reasoning-based inference.

Large language models often retain unintended content, prompting growing interest in knowledge unlearning.Recent approaches emphasize localized unlearning, restricting parameter updates to specific regions in an effort to remove target knowledge while preserving unrelated general knowledge.However, their effectiveness remains uncertain due to the lack of robust and thorough evaluation of the tradeoff between the competing goals of unlearning.In this paper, we begin by revisiting existing localized unlearning approaches.We then conduct controlled experiments to rigorously evaluate whether local parameter updates causally contribute to unlearning.Our findings reveal that the set of parameters that must be modified for effective unlearning is not strictly determined, challenging the core assumption of localized unlearning that parameter locality is inherently indicative of effective knowledge removal.We release our code at https: //github.com/HYU-NLP/loc-unlearn

Conversational agents have traditionally been developed for either task-oriented dialogue (TOD) or open-ended chitchat, with limited progress in unifying the two.Yet, real-world conversations naturally involve fluid transitions between these modes.To address this gap, we introduce TACT (TOD-And-Chitchat Transition), a dataset designed for transitionaware dialogue modeling that incorporates structurally diverse and integrated mode flows.TACT supports both user-and agent-driven mode switches, enabling robust modeling of complex conversational dynamics.To evaluate an agent's ability to initiate and recover from mode transitions, we propose two new metrics-Switch and Recovery.Models trained on TACT outperform baselines in both intent detection and mode transition handling.Moreover, applying Direct Preference Optimization (DPO) to TACT-trained models yields additional gains, achieving 75.74% joint mode-intent accuracy and a 70.1% win rate against GPT-4O in human evaluation.These results demonstrate that pairing structurally diverse data with DPO enhances response quality and transition control, paving the way for more proactive and transition-aware conversational agents.

Entity linking (EL) aligns textual mentions with their corresponding entities in a knowledge base, facilitating various applications such as semantic search and question answering.Recent advances in multimodal entity linking (MEL) have shown that combining text and images can reduce ambiguity and improve alignment accuracy.However, most existing MEL methods overlook the rich structural information available in the form of knowledge-graph (KG) triples.In this paper, we propose KGMEL, a novel framework that leverages KG triples to enhance MEL.Specifically, it operates in three stages: (1) Generation: Produces high-quality triples for each mention by employing vision-language models based on its text and images.(2) Retrieval: Learns joint mention-entity representations, via contrastive learning, that integrate text, images, and (generated or KG) triples to retrieve candidate entities for each mention.(3) Reranking: Refines the KG triples of the candidate entities and employs large language models to identify the bestmatching entity for the mention.Extensive experiments on benchmark datasets demonstrate that KGMEL outperforms existing methods.Our code, datasets, and online appendix are available at: https: //github.com/juyeonnn/KGMEL.

Fine-tuning pre-trained language models (PLMs) has recently shown a potential to improve knowledge graph completion (KGC). However, most PLM-based methods focus solely on encoding textual information, neglecting the long-tailed nature of knowledge graphs and their various topological structures, e.g., subgraphs, shortest paths, and degrees. We claim that this is a major obstacle to achieving higher accuracy of PLMs for KGC. To this end, we propose a Subgraph-Aware Training framework for KGC (SATKGC) with two ideas: (i) subgraph-aware mini-batching to encourage hard negative sampling and to mitigate an imbalance in the frequency of entity occurrences during training, and (ii) new contrastive learning to focus more on harder in-batch negative triples and harder positive triples in terms of the structural properties of the knowledge graph. To the best of our knowledge, this is the first study to comprehensively incorporate the structural inductive bias of the knowledge graph into fine-tuning PLMs. Extensive experiments on three KGC benchmarks demonstrate the superiority of SATKGC. Our code is available.https://github.com/meaningful96/SATKGC

The successful adaptation of multilingual language models (LMs) to a specific languagetask pair critically depends on the availability of data tailored for that condition.While cross-lingual transfer (XLT) methods have contributed to addressing this data scarcity problem, there still exists ongoing debate about the mechanisms behind their effectiveness.In this work, we focus on one of the promising assumptions about the inner workings of XLT, that it encourages multilingual LMs to place greater emphasis on language-agnostic or taskspecific features.We test this hypothesis by examining how the patterns of XLT change with a varying number of source languages involved in the process.Our experimental findings show that the use of multiple source languages in XLT-a technique we term Multi-Source Language Training (MSLT)-leads to increased mingling of embedding spaces for different languages, supporting the claim that XLT benefits from making use of language-independent information.On the other hand, we discover that using an arbitrary combination of source languages does not always guarantee better performance.We suggest simple heuristics for identifying effective language combinations for MSLT and empirically prove its effectiveness.

While the introduction of contrastive learning frameworks in sentence representation learning has significantly contributed to advancements in the field, it still remains unclear whether state-of-the-art sentence embeddings can capture the fine-grained semantics of sentences, particularly when conditioned on specific perspectives.In this paper, we introduce Hyper-CL, an efficient methodology that integrates hypernetworks with contrastive learning to compute conditioned sentence representations.In our proposed approach, the hypernetwork is responsible for transforming pre-computed condition embeddings into corresponding projection layers.This enables the same sentence embeddings to be projected differently according to various conditions.Evaluation of two representative conditioning benchmarks, namely conditional semantic text similarity and knowledge graph completion, demonstrates that Hyper-CL is effective in flexibly conditioning sentence representations, showcasing its computational efficiency at the same time.We also provide a comprehensive analysis of the inner workings of our approach, leading to a better interpretation of its mechanisms.Our code is available at https://github.com/HYU-NLP/Hyper-CL.

Task-oriented dialogue (TOD) systems are commonly designed with the presumption that each utterance represents a single intent. However, this assumption may not accurately reflect real-world situations, where users frequently express multiple intents within a single utterance. While there is an emerging interest in multi-intent detection (MID), existing in-domain datasets such as MixATIS and MixSNIPS have limitations in their formulation. To address these issues, we present BlendX, a suite of refined datasets featuring more diverse patterns than their predecessors, elevating both its complexity and diversity. For dataset construction, we utilize both rule-based heuristics as well as a generative tool—OpenAI’s ChatGPT—which is augmented with a similarity-driven strategy for utterance selection. To ensure the quality of the proposed datasets, we also introduce three novel metrics that assess the statistical properties of an utterance related to word count, conjunction use, and pronoun usage. Extensive experiments on BlendX reveal that state-of-the-art MID models struggle with the challenges posed by the new datasets, highlighting the need to reexamine the current state of the MID field. The dataset is available at https://github.com/HYU-NLP/BlendX.

Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo Lee, Taeuk Kim. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Through in-context learning (ICL), large-scale language models are effective few-shot learners without additional model fine-tuning. However, the ICL performance does not scale well with the number of available training sample as it is limited by the inherent input length constraint of the underlying language model. Meanwhile, many studies have revealed that language models are also powerful feature extractors, allowing them to be utilized in a black-box manner and enabling the linear probing paradigm, where lightweight discriminators are trained on top of the pre-extracted input representations. This paper proposes prompt-augmented linear probing (PALP), a hybrid of linear probing and ICL, which leverages the best of both worlds. PALP inherits the scalability of linear probing and the capability of enforcing language models to derive more meaningful representations via tailoring input into a more conceivable form. Throughout in-depth investigations on various datasets, we verified that PALP significantly closes the gap between ICL in the data-hungry scenario and fine-tuning in the data-abundant scenario with little training overhead, potentially making PALP a strong alternative in a black-box scenario.

Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) is a recent paradigm that attempts to induce constituency parse trees relying only on the internal knowledge of pre-trained language models. While attractive in the perspective that similar to in-context learning, it does not require task-specific fine-tuning, the practical effectiveness of such an approach still remains unclear, except that it can function as a probe for investigating language models’ inner workings. In this work, we mathematically reformulate CPE-PLM and propose two advanced ensemble methods tailored for it, demonstrating that the new parsing paradigm can be competitive with common unsupervised parsers by introducing a set of heterogeneous PLMs combined using our techniques. Furthermore, we explore some scenarios where the trees generated by CPE-PLM are practically useful. Specifically, we show that CPE-PLM is more effective than typical supervised parsers in few-shot settings.

Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Taeuk Kim. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Taeuk Kim, Kang Min Yoo, Sang-goo Lee. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Most existing recursive neural network (RvNN) architectures utilize only the structure of parse trees, ignoring syntactic tags which are provided as by-products of parsing. We present a novel RvNN architecture that can provide dynamic compositionality by considering comprehensive syntactic information derived from both the structure and linguistic tags. Specifically, we introduce a structure-aware tag representation constructed by a separate tag-level tree-LSTM. With this, we can control the composition function of the existing wordlevel tree-LSTM by augmenting the representation as a supplementary input to the gate functions of the tree-LSTM. In extensive experiments, we show that models built upon the proposed architecture obtain superior or competitive performance on several sentence-level tasks such as sentiment analysis and natural language inference when compared against previous tree-structured models and other sophisticated neural models.

We present a latent variable model for predicting the relationship between a pair of text sequences.Unlike previous auto-encodingbased approaches that consider each sequence separately, our proposed framework utilizes both sequences within a single model by generating a sequence that has a given relationship with a source sequence.We further extend the cross-sentence generating framework to facilitate semi-supervised training.We also define novel semantic constraints that lead the decoder network to generate semantically plausible and diverse sequences.We demonstrate the effectiveness of the proposed model from quantitative and qualitative experiments, while achieving state-of-the-art results on semi-supervised natural language inference and paraphrase identification.

We propose a method of stacking multiple long short-term memory (LSTM) layers for modeling sentences. In contrast to the conventional stacked LSTMs where only hidden states are fed as input to the next layer, the suggested architecture accepts both hidden and memory cell states of the preceding layer and fuses information from the left and the lower context using the soft gating mechanism of LSTMs. Thus the architecture modulates the amount of information to be delivered not only in horizontal recurrence but also in vertical connections, from which useful features extracted from lower layers are effectively conveyed to upper layers. We dub this architecture Cell-aware Stacked LSTM (CAS-LSTM) and show from experiments that our models bring significant performance gain over the standard LSTMs on benchmark datasets for natural language inference, paraphrase detection, sentiment classification, and machine translation. We also conduct extensive qualitative analysis to understand the internal behavior of the suggested approach.

Kang Min Yoo, Taeuk Kim, Sang-goo Lee. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.