Pre-trained language models (PLMs) have been successfully used to build high-performance ranking models for large-scale information retrieval systems. However, traditional PLM-based ranking approaches face two key challenges: (1) these models use both sparse and dense content (such as the query/title and content of documents) as inputs, which may require different attention allocations; and (2) traditional PLM-based ranking approaches have identified multiple objectives to gauge user satisfaction with ranking results, but integrating these objectives into the end-to-end training process and the subsequent feature updates and iterations usually involves significant computational resource overhead. In this paper, we propose a novel PLM-based ranking approach M2oE Rank, Multi-objective Mixture-of-Experts (MoE) enhanced Ranking. Specifically, M2oERank lever-ages a context-aware PLM-based hierarchical encoder to extract semantic relevance between the query and the document title and content, while allowing for separate dense and sparse attention for different inputs. With the extracted semantic relevance repre-sentations, multifacet user satisfaction features and task-specific annotations, M2oERank employs an MoE module to perform multi-objective pre-training of ranking models focused on user satisfaction. Finally, M2oERank uses a weight fusion module that fuses outputs from the above experts to predict ranking scores. Moreover, we present a three-stage offline training strategy and the online system workflow for deploying M2oERank at web-scale search. To demonstrate the effectiveness of our proposed approach, we conduct extensive offline and online evaluations using real-world web traffic from Baidu Search. The comparisons against numbers of advanced baselines confirmed the advantages of M2oERank in producing high-performance ranking models for web-scale search.

As modern learning-to-rank (LTR) systems rely on both textual and behavioral features, it is essential to extend pre-trained language models (PLMs) from text (queries and webpages) understanding to end-to-end ranking score prediction subject to multiple objectives, such as relevance, quality, authority, and recency. While textual inputs encompass a broader array of features than mere relevance and behavioral features are frequently skewed by user feedback with position bias, an integrated solution is required to jointly disentangle and fuse these heterogeneous features, ensuring robust and unbiased ranking predictions. In this work, we introduce RankExpert, a unified framework that holistically models heterogeneous ranking signals by integrating PLM-based semantic extraction with behavioral cues. RankExpert employs a lightweight PLM with hierarchical distillation for efficient query-document representation, a mixture-of-experts (MoE) architecture to disentangle and optimize objectives across relevance, quality, authority, and recency, and a click expert to mitigate position bias in user feedback within a dual-tower design. Then, RankExpert utilizes an adaptive weight fusion layer to dynamically integrate the outputs of these specialized experts dynamically, aligning ranking predictions with diverse user intents. Extensive offline evaluations on two large-scale real-world datasets reveal that RankExpert significantly outperforms strong competitor models and substantial gains in key performance indicators. Moreover, we deploy RankExpert at Baidu Search and conduct comprehensive online evaluations with real-world web traffic, demonstrating substantial improvements in user satisfaction metrics over the online legacy system and underscoring the practicality of the proposed approach.

The exponential growth of online content and increasingly diverse user needs have underscored the necessity for ranking models that go beyond traditional relevance assessments. Although several open-source benchmarks have significantly advanced academic research in Learning-to-Rank (LTR), these datasets predominantly focus on either text-based relevance or user behavior (click-through or dwell time) signals separately. This separation has inadvertently burdened academic progress by limiting the exploration of multifaceted, satisfaction-oriented ranking models. In contrast, industry research has begun to delve into integrated approaches that fuse prior (relevance, authority, recency, and quality) with posterior (user interaction such as clicks and dwell time) signals, thereby better capturing true user satisfaction. In this paper, we introduce FULTR-a large-scale, prior-posterior FUsion LTR dataset. FULTR comprises over 224M queries and 683M documents from Baidu Search, combining both: (1) a rich prior-attribute set with detailed textual relevance, authority, recency, and quality features, and (2) a comprehensive posterior-attribute set enriched by user click data, dwell time, and positional information. By unifying these dual perspectives, FULTR establishes a robust, reproducible benchmark for satisfaction-oriented ranking, enabling researchers to develop models that better capture real-world search behaviors and user satisfaction. In addition, we propose a strong LTR baseline that merges a satisfaction ranker that leverages pre-trained language models to integrate diverse satisfaction signals, with a behavior ranker that captures user interactions using a dual-tower approach. Their outputs are combined via a fusion layer, yielding significant performance gains in multiple evaluation metrics, as confirmed by extensive experiments and ablation studies. We are confident that our contribution not only democratizes access to industrial-grade fusion data for the research community but also paves the way for more effective and holistic LTR model design. FULTR is available to the research community at https://github.com/zhanghao731/FULTR.