Large Language Model-based Dense Retrieval (LLM-DR) optimizes over numerous heterogeneous fine-tuning collections from different domains. However, the discussion about its training data distribution is still minimal. Previous studies rely on empirically assigned dataset choices or sampling ratios, which inevitably lead to sub-optimal retrieval performances. In this paper, we propose a new task-level Distributionally Robust Optimization (tDRO) algorithm for LLM-DR fine-tuning, targeted at improving the universal domain generalization ability by end-to-end reweighting the data distribution of each task. The tDRO parameterizes the domain weights and updates them with scaled domain gradients. The optimized weights are then transferred to the LLM-DR fine-tuning to train more robust retrievers. Experiments show optimal improvements in large-scale retrieval benchmarks and reduce up to 30% dataset usage after applying our optimization algorithm with a series of different-sized LLM-DR models.

Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Zhenpeng Su, Xing W, Wei Zhou, Guangyuan Ma, Songlin Hu. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Masked auto-encoder pre-training has emerged as a prevalent technique for initializing and enhancing dense retrieval systems. It generally utilizes additional Transformer decoder blocks to provide sustainable supervision signals and compress contextual information into dense representations. However, the underlying reasons for the effectiveness of such a pre-training technique remain unclear. The usage of additional Transformer-based decoders also incurs significant computational costs. In this study, we aim to shed light on this issue by revealing that masked auto-encoder (MAE) pre-training with enhanced decoding significantly improves the term coverage of input tokens in dense representations, compared to vanilla BERT checkpoints. Building upon this observation, we propose a modification to the traditional MAE by replacing the decoder of a masked auto-encoder with a completely simplified Bag-of-Word prediction task. This modification enables the efficient compression of lexical signals into dense representations through unsupervised pre-training. Remarkably, our proposed method achieves state-of-the-art retrieval performance on several large-scale retrieval benchmarks without requiring any additional parameters, which provides a 67% training speed-up compared to standard masked auto-encoder pre-training with enhanced decoding.

Dense passage retrieval aims to retrieve the relevant passages of a query from a large corpus based on dense representations (i.e., vectors) of the query and the passages. Recent studies have explored improving pre-trained language models to boost dense retrieval performance. This paper proposes CoT-MAE (ConTextual Masked Auto-Encoder), a simple yet effective generative pre-training method for dense passage retrieval. CoT-MAE employs an asymmetric encoder-decoder architecture that learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding. Precisely, self-supervised masked auto-encoding learns to model the semantics of the tokens inside a text span, and context-supervised masked auto-encoding learns to model the semantical correlation between the text spans. We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines, demonstrating the high efficiency of CoT-MAE. Our code is available at https://github.com/caskcsg/ir/tree/main/cotmae.

Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training.These methods simply consider two passages from the same document to be relevant, without taking into account the potential negative impacts of weakly correlated pairs.Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue.Query-as-context pretraining assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair.These passage-query pairs are then used in contrastive or generative context-supervised pre-training.The pre-trained models are evaluated on largescale passage retrieval benchmarks and out-ofdomain zero-shot benchmarks.Experimental results show that query-as-context pre-training brings considerable gains for retrieval performances, demonstrating its effectiveness and efficiency.

The coronavirus disease 2019 (COVID-19) pandemic, caused by the coronavirus severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has created an unprecedented threat to public health. The pandemic has been sweeping the globe, impacting more than 200 countries, with more outbreaks still lurking on the horizon. At the time of the writing, no approved drugs or vaccines are available to treat COVID-19 patients, prompting an urgent need to decipher mechanisms underlying the pathogenesis and develop curative treatments. To fight COVID-19, researchers around the world have provided specific tools and molecular information for SARS-CoV-2. These pieces of information can be integrated to aid computational investigations and facilitate clinical research. This paper reviews current knowledge, the current status of drug development and various resources for key steps toward effective treatment of COVID-19, including the phylogenetic characteristics, genomic conservation and interaction data. The final goal of this paper is to provide information that may be utilized in bioinformatics approaches and aid target prioritization and drug repurposing. Several SARS-CoV-2-related tools/databases were reviewed, and a web-portal named OverCOVID (http://bis.zju.edu.cn/overcovid/) is constructed to provide a detailed interpretation of SARS-CoV-2 basics and share a collection of resources that may contribute to therapeutic advances. These information could improve researchers' understanding of SARS-CoV-2 and help to accelerate the development of new antiviral treatments.