Reasoning-intensive ranking models built on Large Language Models (LLMs) have made notable progress. However, existing approaches often rely on large-scale LLMs and explicit Chain-of-Thought (CoT) reasoning, resulting in high computational cost and latency that limit real-world use. To address this, we propose TFRank, an efficient pointwise reasoning ranker based on small-scale LLMs. To improve ranking performance, TFRank effectively integrates CoT data, fine-grained score supervision, and multi-task training. Furthermore, it achieves an efficient "Think-Free" reasoning capability by employing a "think-mode switch" and pointwise format constraints. Specifically, this allows the model to leverage explicit reasoning during training while delivering precise relevance scores for complex queries at inference without generating any reasoning chains. Experiments show that TFRank achieves performance comparable to models with four times more parameters on the BRIGHT benchmark, and demonstrates strong competitiveness on the BEIR benchmark. Further analysis shows that TFRank achieves an effective balance between performance and efficiency, providing a practical solution for integrating advanced reasoning into real-world systems.

As a core module of web search, query understanding aims to bridge the semantic gap between user queries and web page documents, thereby enhancing the ability to deliver more relevant results.Recently, Large Language Models (LLMs) have achieved significant breakthroughs that have fundamentally altered the workflow of existing search ranking tasks.However, few researchers have explored the integration of LLMs into the field of query understanding.In this paper, we investigate the potential of LLMs in query understanding by conducting a comprehensive evaluation across three dimensions: term, structure, and topic.This evaluation includes several representative tasks such as segmentation, term weighting, error correction, query expansion, and intent recognition.The experimental results reveal that LLMs are particularly effective in query expansion and intent recognition but show limited improvement in other areas.This limitation may be attributed to LLMs' primary focus on modeling the semantic knowledge of entire queries, while lacking the capability to capture token-level information with finer granularity.Additionally, we explore potential practical applications of LLMs in query understanding, such as integrating the evaluation and training capabilities of smaller models with LLMs and constructing unsupervised samples.Based on comprehensive empirical results, collaborative training emerges as a promising approach to leverage LLMs for query understanding.We hope this research will advance the practical application of LLMs in query understanding and contribute to the development of this field.

Relevance plays a crucial role in commercial search engines by identifying documents related to user queries and fulfilling their search needs.Traditional approaches employ encoder-only models like BERT, which process concatenated query-document pairs to predict relevance scores.While autoregressive large language models (LLMs) have revolutionized numerous NLP domains, their direct application to web-scale search systems presents significant challenges.On one hand, the relevance modeling capabilities of LLMs have not been fully explored.On the other, the high computational costs and inference times make deploying LLMs in online search systems, which demand extremely low latency, nearly impossible.In this work, we address these challenges through two key contributions.First, we develop a comprehensive evaluation framework to systematically assess the effectiveness of LLMs in query-document relevance ranking.By conducting assessment experiments to LLMs in four perspectives: ranking objectives, model size, domain-specific continuous pre-training, and the integration of prior knowledge, we identify the best resource allocation strategy given a restricted budget and develop practical LLMs in a more efficient way.Second, we propose a novel framework to transfer the capabilities of LLMs in the ranking aspect to existing BERT models to avoid directly deploying LLMs.Finally, to fully leverage the improvements in relevance ranking brought by LLMs, we successfully nearline deploy LLMs in Tencent QQ Browser search engine using query-based ondemand computing and quantization.Experiments on real-world datasets and online A/B tests demonstrate that our approach significantly enhances search engine performance while maintaining practical operational efficiency.Our findings provide actionable insights for integrating LLMs into production search engines.

Haolin Deng, Chang Wang, Li Xin, Dezhang Yuan, Junlang Zhan, Tian Zhou, Jin Ma, Jun Gao, Ruifeng Xu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

This paper addresses the challenge of the semantic gap between user queries and web content, commonly referred to as asymmetric text matching, within the domain of web search.By leveraging BERT for reading comprehension, current algorithms enable significant advancements in query understanding, but still encounter limitations in effectively resolving the asymmetrical ranking problem due to model comprehension and summarization constraints.To tackle this issue, we propose the QAGR (Question-Answer Generation and Ranking) method, comprising an offline module called QAGeneration and an online module called QARanking.The QAGeneration module utilizes large language models (LLMs) to generate high-quality question-answering pairs for each web page.This process involves two steps: generating question-answer pairs and performing verification to eliminate irrelevant questions, resulting in high-quality questions associated with their respective documents.The QARanking module combines and ranks the generated questions and web page content.To ensure efficient online inference, we design the QARanking model as a homogeneous dual-tower model, incorporating query intent to drive score fusion while balancing keyword matching and asymmetric matching.Additionally, we conduct a preliminary screening of questions for each document, selecting only the top-N relevant questions for further relevance calculation.Empirical results demonstrate the substantial performance improvement of our proposed method in web search.We achieve over

Vision-language alignment learning for video-text retrieval arouses a lot of attention in recent years. Most of the existing methods either transfer the knowledge of image-text pretraining model to video-text retrieval task without fully exploring the multi-modal information of videos, or simply fuse multi-modal features in a brute force manner without explicit guidance. In this paper, we integrate multi-modal information in an explicit manner by tagging, and use the tags as the anchors for better video-text alignment. Various pretrained experts are utilized for extracting the information of multiple modalities, including object, person, motion, audio, etc. To take full advantage of these information, we propose the TABLE (TAgging Before aLignmEnt) network, which consists of a visual encoder, a tag encoder, a text encoder, and a tag-guiding cross-modal encoder for jointly encoding multi-frame visual features and multi-modal tags information. Furthermore, to strengthen the interaction between video and text, we build a joint cross-modal encoder with the triplet input of [vision, tag, text] and perform two additional supervised tasks, Video Text Matching (VTM) and Masked Language Modeling (MLM). Extensive experimental results demonstrate that the TABLE model is capable of achieving State-Of-The-Art (SOTA) performance on various video-text retrieval benchmarks, including MSR-VTT, MSVD, LSMDC and DiDeMo.

SongYang Gao, Shihan Dou, Yan Liu, Xiao Wang, Qi Zhang, Zhongyu Wei, Jin Ma, Ying Shan. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

In the real-world scenario, a longstanding goal of multilingual neural machine translation (MNMT) is that a single model can incrementally adapt to new language pairs without accessing previous training data.In this scenario, previous studies concentrate on overcoming catastrophic forgetting while lacking encouragement to learn new knowledge from incremental language pairs, especially when the incremental language is not related to the set of original languages.To better acquire new knowledge, we propose a knowledge transfer method that can efficiently adapt original MNMT models to diverse incremental language pairs.The method flexibly introduces the knowledge from an external model into original models, which encourages the models to learn new language pairs, completing the procedure of knowledge transfer.Moreover, all original parameters are frozen to ensure that translation qualities on original language pairs are not degraded.Experimental results show that our method can learn new knowledge from diverse language pairs incrementally meanwhile maintaining performance on original language pairs, outperforming various strong baselines in incremental learning for MNMT.

Query correction is a task that automatically detects and corrects errors in what users type into a search engine. Misspelled queries can lead to user dissatisfaction and churn. However, correcting a user query accurately is not an easy task. One major challenge is that a correction model must be capable of high-level language comprehension. Recently, pre-trained language models (PLMs) have been successfully applied to text correction tasks, but few works have been done on query correction. However, it is nontrivial to directly apply these PLMs to query correction in large-scale search systems due to the following challenging issues: 1) Expensive deployment. Deploying such a model requires expensive computations. 2) Lacking domain knowledge. A neural correction model needs massive training data to activate its power.

The rapid explosion of video distribution is accompanied by a massive amount of video text, which encompasses rich information about the video content. While previous research has primarily focused on text extraction from videos like text detection, tracking, recognition and end to end spotting, the layout of video text has received limited attention. As different text categories convey distinct meanings, video text layout is critical for video understanding tasks such as video summarization and shooting environment comprehension. To bridge the gap between video OCR and understanding, we explore the study of video text layout in this work. We first optimize the layout annotation of the BOVText, a bilingual, open-world video text dataset, by expanding text categories and defining five clear categories: scene, subtitle, title, logo, and other. Additionally, we rectify the original unreasonable layout annotation based on these definitions. We also propose a Video-level Text Layout model (VTLayout) to address the layout problem, which fuses textual, visual, and spatial-temporal embedding of video text trajectories. To the best of our knowledge, this is the first method to tackle text layout on video level. Our method outperforms image-level layout methods across all text categories and exhibits faster inference speed. This study underscores the significance of video text layout in video understanding and offers an effective solution to this challenge. Our annotation is available at https://github.com/TencentARC/VTLayout.

Passage ranking involves two stages: passage retrieval and passage re-ranking, which are important and challenging topics for both academics and industries in the area of Information Retrieval (IR). However, the commonly-used datasets for passage ranking usually focus on the English language. For non-English scenarios, such as Chinese, the existing datasets are limited in terms of data scale, fine-grained relevance annotation and false negative issues. To address this problem, we introduce T2Ranking, a large-scale Chinese benchmark for passage ranking. T2Ranking comprises more than 300K queries and over 2M unique passages from real-world search engines. Expert annotators are recruited to provide 4-level graded relevance scores (fine-grained) for query-passage pairs instead of binary relevance judgments (coarse-grained). To ease the false negative issues, more passages with higher diversities are considered when performing relevance annotations, especially in the test set, to ensure a more accurate evaluation. Apart from the textual query and passage data, other auxiliary resources are also provided, such as query types and XML files of documents which passages are generated from, to facilitate further studies. To evaluate the dataset, commonly used ranking models are implemented and tested on T2Ranking as baselines. The experimental results show that T2Ranking is challenging and there is still scope for improvement. The full data and all codes are available at https://github.com/THUIR/T2Ranking/.

Visual question answering is an important task in both natural language and vision understanding. However, in most of the public visual question answering datasets such as VQA, CLEVR, the questions are human generated that specific to the given image, such as 'What color are her eyes?'. The human generated crowdsourcing questions are relatively simple and sometimes have the bias toward certain entities or attributes [1, 55].

Haolin Deng, Yanan Zhang, Yangfan Zhang, Wangyang Ying, Changlong Yu, Jun Gao, Wei Wang, Xiaoling Bai, Nan Yang, Jin Ma, Xiang Chen, Tianhua Zhou. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

In a practical real-world scenario, the longstanding goal is that a universal multilingual translation model can be incrementally updated when new language pairs arrive.Specifically, the initial vocabulary only covers some of the words in new languages, which hurts the translation quality for incremental learning.Although existing approaches attempt to address this issue by replacing the original vocabulary with a rebuilt vocabulary or constructing independent language-specific vocabularies, these methods can not meet the following three demands simultaneously: (1) High translation quality for original and incremental languages, (2) low cost for model training, (3) low time overhead for preprocessing.In this work, we propose an entropy-based vocabulary substitution (EVS) method that just needs to walk through new language pairs for incremental learning in a large-scale multilingual data updating while remaining the size of the vocabulary.Our method has access to learn new knowledge from updated training samples incrementally while keeping high translation quality for original language pairs, alleviating the issue of catastrophic forgetting.Results of experiments show that EVS can achieve better performance and save excess overhead for incremental learning in the multilingual machine translation task.