Query performance prediction (QPP) is a critical task in information retrieval. It estimates retrieval quality for a given query without relying on relevance judgments. While recent approaches have leveraged pretrained (large) language models with binary- or cross-encoder architectures, they struggle to capture subtle semantic differences (nuances that make similar sentences mean different things) between queries and documents in QPP, limiting prediction accuracy. To address this issue, we present Angle-QPP, a novel and efficient binary-encoder QPP approach with three key innovations: (1) the use of Large Language Models (LLMs) of varying scales to learn rich contextual semantics, (2) a contrastive learning warm-up phase to obtain high-quality initial representation quality, and (3) an angle-based interaction mechanism operating in complex embedding space to effectively capture subtle semantic relationships between queries and documents. Comprehensive experiments on TREC DL 2019, 2020, 2021, and 2022 datasets demonstrate that the proposed Angle-QPP significantly outperforms existing methods across all evaluation metrics. Notably, Angle-QPP models with 0.5B, 1.5B, and 3B parameters achieve \(6.4\%\) , \(11.2\%\) , and \(13.9\%\) absolute improvements in prediction accuracy over the previous state-of-the-art binary-encoder BERT-QPP, respectively. It demonstrates the scalability and effectiveness of the proposed method. Ablation studies confirm the effectiveness of both the angle interaction mechanism and contrastive learning warm-up components. Our analysis further reveals that scaling up LLM size consistently improves QPP performance, providing valuable insights for the design of future QPP systems.

Gao Zuchen, Zizheng Zhan, Xianming Li, Erxin Yu, Haotian Zhang, Chenbin Chenbin, Yuqun Zhang, Jing Li. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

High-quality sentence embeddings are fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval-augmented generation (RAG). However, most existing methods leverage fixed-length sentence embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various applications. Viewing this gap, we propose a novel sentence embedding model Espresso Sentence Embeddings (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to shallow layers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way, ESE can scale model depth via the former process and embedding size via the latter. Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality sentence embeddings with less model depth and embedding size, enhancing inference efficiency. The code is available at https://github.com/SeanLee97/AnglE/blob/main/README_ESE.md.

Text embedding is pivotal in semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications.STS learning largely relies on the cosine function as the optimization objective to reflect semantic similarity.However, the cosine has saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeddings.To address this issue, we propose a novel Angle-optimized Embedding model, AoE.It optimizes angle differences in complex space to explore similarity in saturation zones better.To set up a comprehensive evaluation, we experimented with existing short-text STS, our newly collected long-text STS, and downstream task datasets.Extensive experimental results on STS and MTEB benchmarks show that AoE significantly outperforms popular text embedding models neglecting cosine saturation zones.It highlights that AoE can produce highquality text embeddings and broadly benefit downstream tasks.The code is available at:

Sentence embeddings are crucial in measuring semantic similarity.Most recent studies employed large language models (LLMs) to learn sentence embeddings.Existing LLMs mainly adopted autoregressive architecture without explicit backward dependency modeling.Therefore, we examined the effects of backward dependencies in LLMs for semantic similarity measurements.Concretely, we propose a novel model: backward dependency enhanced large language model (BeLLM).It learns sentence embeddings via transforming specific attention layers from uni-to bi-directional.We extensively experiment across various semantic textual similarity (STS) tasks and downstream applications.BeLLM achieves state-of-the-art performance in varying scenarios.It shows that auto-regressive LLMs benefit from backward dependencies for sentence embeddings. 1

Researchers have been aware that emotion is not one-hot encoded in emotion-relevant classification tasks, and multiple emotions can coexist in a given sentence. Recently, several works have focused on leveraging a distribution label or a grayscale label of emotions in the classification model, which can enhance the one-hot label with additional information, such as the intensity of other emotions and the correlation between emotions. Such an approach has been proven effective in alleviating the overfitting problem and improving the model robustness by introducing a distribution learning component in the objective function. However, the effect of distribution learning cannot be fully unfolded as it can reduce the model’s discriminative ability within similar emotion categories. For example, “Sad” and “Fear” are both negative emotions. To address such a problem, we proposed a novel emotion extension scheme in the prior work (Li, Chen, Xie, Li, and Tao, 2021). The prior work incorporated fine-grained emotion concepts to build an extended label space, where a mapping function between coarse-grained emotion categories and fine-grained emotion concepts was identified. For example, sentences labeled “Joy” can convey various emotions such as enjoy, free, and leisure. The model can further benefit from the extended space by extracting dependency within fine-grained emotions when yielding predictions in the original label space. The prior work has shown that it is more apt to apply distribution learning in the extended label space than in the original space. A novel sparse connection method, i.e., Leaky Dropout, is proposed in this paper to refine the dependency-extraction step, which further improves the classification performance. In addition to the multiclass emotion classification task, we extensively experimented on sentiment analysis and multilabel emotion prediction tasks to investigate the effectiveness and generality of the label extension schema.

Currently, text classification studies mainly focus on training classifiers by using textual input only, or enhancing semantic features by introducing external knowledge (e.g., hand-craft lexicons and domain knowledge). In contrast, some intrinsic statistical features of the corpus, like word frequency and distribution over labels, are not well exploited. Compared with external knowledge, the statistical features are deterministic and naturally compatible with corresponding tasks. In this paper, we propose an Adaptive Gate Network (AGN) to consolidate semantic representation with statistical features selectively. In particular, AGN encodes statistical features through a variational component and merges information via a well-designed valve mechanism. The valve adapts the information flow into the classifier according to the confidence of semantic features in decision making, which can facilitate training a robust classifier and can address the overfitting caused by using statistical features. Extensive experiments on datasets of various scales show that, by incorporating statistical information, AGN can improve the classification performance of CNN, RNN, Transformer, and Bert based models effectively. The experiments also indicate the robustness of AGN against adversarial attacks of manipulating statistical information.

Joint extraction of entities and relations from unstructured texts to form factual triples is a fundamental task of constructing a Knowledge Base (KB).A common method is to decode triples by predicting entity pairs to obtain the corresponding relation.However, it is still challenging to handle this task efficiently, especially for the overlapping triple problem.To address such a problem, this paper proposes a novel efficient entities and relations extraction model called TDEER, which stands for Translating Decoding Schema for Joint Extraction of Entities and Relations.Unlike the common approaches, the proposed translating decoding schema regards the relation as a translating operation from subject to objects, i.e., TDEER decodes triples as subject + relation → objects.TDEER can naturally handle the overlapping triple problem, because the translating decoding schema can recognize all possible triples, including overlapping and non-overlapping triples.To enhance model robustness, we introduce negative samples to alleviate error accumulation at different stages.Extensive experiments on public datasets demonstrate that TDEER produces competitive results compared with the state-of-the-art (SOTA) baselines.Furthermore, the computation complexity analysis indicates that TDEER is more efficient than powerful baselines.Especially, the proposed TDEER is 2 times faster than the recent SOTA models.The code is available at https://github.com/4AI/TDEER.

Semantic segmentation is a challenging problem in computer vision. Many applications, such as autonomous driving and robot navigation with urban road scene, need accurate and efficient segmentation. Most state-of-the-art methods focus on accuracy, rather than efficiency. In this paper, we propose a more efficient neural network architecture, which has fewer parameters, for semantic segmentation in the urban road scene. An asymmetric encoder-decoder structure based on ResNet is used in our model. In the first stage of encoder, we use continuous factorized block to extract low-level features. Continuous dilated block is applied in the second stage, which ensures that the model has a larger view field, while keeping the model small-scale and shallow. The down sampled features from encoder are up sampled with decoder to the same-size output as the input image and the details refined. Our model can achieve end-to-end and pixel-to-pixel training without pretraining from scratch. The parameters of our model are only 0.2M, 100× less than those of others such as SegNet, etc. Experiments are conducted on five public road scene datasets (CamVid, CityScapes, Gatech, KITTI Road Detection, and KITTI Semantic Segmentation), and the results demonstrate that our model can achieve better performance.