Large Language Models (LLMs) have shown strong capabilities in open-domain question answering (QA), but deploying them in real-world online systems introduces critical challenges. These include: (1) handling both simple and complex queries with appropriate levels of reasoning, (2) minimizing latency without compromising answer quality, and (3) maintaining answer consistency under evolving and noisy retrieval contexts. To address these challenges, we propose Retain-to-Refine (ℜ2ℜ), an adaptive agent-based QA framework designed for practical deployment. ℜ2ℜ integrates a Query Critic Agent (QCA) to assess query difficulty and route it accordingly: simple queries are answered directly using fast, prompt-based LLM calls, while complex queries are handled by a Memory Augmented Agent (MAA). MAA performs iterative reasoning guided by a unique long-short memory mechanism. Long-term memory retains and consolidates stable, core facts to ground the reasoning process, while short-term memory identifies transient information gaps to formulate highly focused subsequent queries. To ensure evidence quality, a Supervised Retrospection module validates and filters retrieved documents at each step. This agent-based design enables ℜ2ℜ to dynamically allocate computation based on question complexity, reducing unnecessary overhead while preserving high-quality answers when multi-step reasoning or external knowledge is required. Extensive evaluations across various settings and datasets demonstrate that the efficiency of R2R across diverse question types. In online settings, ℜ2ℜ delivers substantial gains in both response quality and efficiency, making it well-suited for large-scale industrial deployment in real-time QA services.

While agentic AI systems rely on LLMs to translate user intent into structured function calls, this process is fraught with computational redundancy, leading to high inference latency that hinders real-time applications. This paper identifies and addresses three key redundancies: (1) the redundant processing of a large library of function descriptions for every request; (2) the redundant use of a large, slow model to generate an entire, often predictable, token sequence; and (3) the redundant generation of fixed, boilerplate parameter syntax. We introduce HyFunc, a novel framework that systematically eliminates these inefficiencies. HyFunc employs a hybrid-model cascade where a large model distills user intent into a single ''soft token.'' This token guides a lightweight retriever to select relevant functions and directs a smaller, prefix-tuned model to generate the final call, thus avoiding redundant context processing and full-sequence generation by the large model. To eliminate syntactic redundancy, our ''dynamic templating'' technique injects boilerplate parameter syntax on-the-fly within an extended vLLM engine. To avoid potential limitations in generalization, we evaluate HyFunc on an unseen benchmark dataset, BFCL. Experimental results demonstrate that HyFunc achieves an excellent balance between efficiency and performance. It achieves an inference latency of 0.828 seconds, outperforming all baseline models, and reaches a performance of 80.1%, surpassing all models with a comparable parameter scale. These results suggest that HyFunc offers a more efficient paradigm for agentic AI. Our code is publicly available at https://github.com/MrBlankness/HyFunc.

Generative inference with Large Language Models (LLMs) is the cornerstone of web-scale AI search, where queries are answered using vast, heterogeneous documents retrieved via Retrieval-Augmented Generation (RAG). This paradigm is critically bottlenecked by the cost of self-attention mechanism on long context. The sheer diversity of retrieved web content (multi-sourced, multi-lingual, multi-faceted) makes simple Key-Value (KV) cache optimizations with pre-fixed subsets ineffective, demanding a dynamic, content-aware approach. This challenge, however, introduces a classic chicken-and-egg problem: the model cannot foresee the necessary KV entries for attention without first inferring on the content, yet doing so on the full context is prohibitively expensive. This paper introduces P&F, a unified framework that resolves this dilemma through a core ''probe-and-fetch'' mechanism, which ingeniously integrates with speculative decoding -- an acceleration approach already adopted in web-scale AI search. The probe step repurposes the speculative draft model: while generating candidate tokens, it simultaneously probes the context to predict the most salient KV entries the large model will need for attention. The fetch step immediately acts on this prediction, asynchronously fetching these sparse entries from memory. This synergistic design piggybacks the probing step onto the drafting process, allowing the expensive gathering of a sparse KV cache to be fully masked. Crucially, this co-design breaks the sequential dependency bottleneck that cripples naive integrations of speculative decoding and prefetching due to synchronization issues. Extensive experiments show P&F significantly outperforms state-of-the-art methods in throughput and scalability, offering a practical, drop-in solution. Extensive offline evaluations across various settings and datasets demonstrate that P&F yields superior throughput and scalability compared to advanced baselines, while maintaining model quality across diverse models and scales. In online settings, P&F delivers substantial gains in throughput improvements while preserving response quality, making it well-suited for large-scale industrial deployment in real-time AI Search services.

This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state-of-the-art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines what to evaluate and another that explains how to evaluate . The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues. Together, these frameworks summarize the current status quo, expose limitations in traditional practices, and provide a structured blueprint for improvement. Based on the summarization of existing studies, we identify several challenges and propose future directions, including the development of scalable, real-time evaluation pipelines, enhanced privacy-preserving mechanisms, and robust metrics that capture dynamic multi-turn interactions. Our contributions bridge historical insights with modern practices, paving the way for next-generation, reliably evaluated conversational AI systems and offering a comprehensive guide for researchers and practitioners.

Pre-trained language models (PLMs) have been successfully used to build high-performance ranking models for large-scale information retrieval systems. However, traditional PLM-based ranking approaches face two key challenges: (1) these models use both sparse and dense content (such as the query/title and content of documents) as inputs, which may require different attention allocations; and (2) traditional PLM-based ranking approaches have identified multiple objectives to gauge user satisfaction with ranking results, but integrating these objectives into the end-to-end training process and the subsequent feature updates and iterations usually involves significant computational resource overhead. In this paper, we propose a novel PLM-based ranking approach M2oE Rank, Multi-objective Mixture-of-Experts (MoE) enhanced Ranking. Specifically, M2oERank lever-ages a context-aware PLM-based hierarchical encoder to extract semantic relevance between the query and the document title and content, while allowing for separate dense and sparse attention for different inputs. With the extracted semantic relevance repre-sentations, multifacet user satisfaction features and task-specific annotations, M2oERank employs an MoE module to perform multi-objective pre-training of ranking models focused on user satisfaction. Finally, M2oERank uses a weight fusion module that fuses outputs from the above experts to predict ranking scores. Moreover, we present a three-stage offline training strategy and the online system workflow for deploying M2oERank at web-scale search. To demonstrate the effectiveness of our proposed approach, we conduct extensive offline and online evaluations using real-world web traffic from Baidu Search. The comparisons against numbers of advanced baselines confirmed the advantages of M2oERank in producing high-performance ranking models for web-scale search.

As modern learning-to-rank (LTR) systems rely on both textual and behavioral features, it is essential to extend pre-trained language models (PLMs) from text (queries and webpages) understanding to end-to-end ranking score prediction subject to multiple objectives, such as relevance, quality, authority, and recency. While textual inputs encompass a broader array of features than mere relevance and behavioral features are frequently skewed by user feedback with position bias, an integrated solution is required to jointly disentangle and fuse these heterogeneous features, ensuring robust and unbiased ranking predictions. In this work, we introduce RankExpert, a unified framework that holistically models heterogeneous ranking signals by integrating PLM-based semantic extraction with behavioral cues. RankExpert employs a lightweight PLM with hierarchical distillation for efficient query-document representation, a mixture-of-experts (MoE) architecture to disentangle and optimize objectives across relevance, quality, authority, and recency, and a click expert to mitigate position bias in user feedback within a dual-tower design. Then, RankExpert utilizes an adaptive weight fusion layer to dynamically integrate the outputs of these specialized experts dynamically, aligning ranking predictions with diverse user intents. Extensive offline evaluations on two large-scale real-world datasets reveal that RankExpert significantly outperforms strong competitor models and substantial gains in key performance indicators. Moreover, we deploy RankExpert at Baidu Search and conduct comprehensive online evaluations with real-world web traffic, demonstrating substantial improvements in user satisfaction metrics over the online legacy system and underscoring the practicality of the proposed approach.

The exponential growth of online content and increasingly diverse user needs have underscored the necessity for ranking models that go beyond traditional relevance assessments. Although several open-source benchmarks have significantly advanced academic research in Learning-to-Rank (LTR), these datasets predominantly focus on either text-based relevance or user behavior (click-through or dwell time) signals separately. This separation has inadvertently burdened academic progress by limiting the exploration of multifaceted, satisfaction-oriented ranking models. In contrast, industry research has begun to delve into integrated approaches that fuse prior (relevance, authority, recency, and quality) with posterior (user interaction such as clicks and dwell time) signals, thereby better capturing true user satisfaction. In this paper, we introduce FULTR-a large-scale, prior-posterior FUsion LTR dataset. FULTR comprises over 224M queries and 683M documents from Baidu Search, combining both: (1) a rich prior-attribute set with detailed textual relevance, authority, recency, and quality features, and (2) a comprehensive posterior-attribute set enriched by user click data, dwell time, and positional information. By unifying these dual perspectives, FULTR establishes a robust, reproducible benchmark for satisfaction-oriented ranking, enabling researchers to develop models that better capture real-world search behaviors and user satisfaction. In addition, we propose a strong LTR baseline that merges a satisfaction ranker that leverages pre-trained language models to integrate diverse satisfaction signals, with a behavior ranker that captures user interactions using a dual-tower approach. Their outputs are combined via a fusion layer, yielding significant performance gains in multiple evaluation metrics, as confirmed by extensive experiments and ablation studies. We are confident that our contribution not only democratizes access to industrial-grade fusion data for the research community but also paves the way for more effective and holistic LTR model design. FULTR is available to the research community at https://github.com/zhanghao731/FULTR.

The proliferation of complex non-factoid questions in modern information seeking (IS) systems exposes critical limitations in conventional Retrieval-Augmented Generation (RAG) approaches, particularly their static search strategies and the lack of systematic multi-source information integration capabilities. Facing these limitations, we present PASS (Proactive Agent-driven Search System), a novel multi-agent framework that operationalizes human-like proactive search strategies through five specialized agents: Revealer for intent analysis, Navigator for search planning, Seeker/Reader for adaptive retrieval, and Writer for response synthesis, systematically expanding the search space through iterative query refinement and multi-perspective knowledge integration. Crucially, our framework demonstrates remarkable adaptability to mid-sized LLMs, demonstrating its scalability in resource-constrained environments. To comprehensively assess the effectiveness of the proposed framework, we carry out extensive experiments on both mid-sized and proprietary large-scale LLMs, evaluating response quality for complex non-factoid questions using a newly introduced nugget-based assessment. Experimental results from offline nugget-based evaluation and online A/B Tests confirm substantial improvements in answer quality, advancing proactive information seeking methodologies and offering practical pathways for democratizing complex reasoning capabilities to resource-constrained environments.

While representation learning has been used to boost the performance of Learning-to-Rank (LTR) models through distilling key features for webpage ranking, the weak supervision signals extracted from users' sparse click-through data lead to inadequate representation of query-webpage pairs for ranking score prediction. Recent studies in generative LTR pre-training demonstrate the feasibility of incorporating reconstruction loss for enhanced ranking score prediction. However, LTR is afterall a regression task and it might be reasonable to find an alternate route that pre-trains LTR models with discriminative losses. Following the success of Electra in representation learning for natural language processing (NLP), this work proposes RankElectra that pre-trains the LTR model as a discriminator module inside a generative learning framework. Specifically, RankElectra first structures sparsely-annotated query-webpage pairs into a bipartite graph, with query and webpage feature vectors as node types and ranking scores as the connecting edges, and then leverages positive and negative extension strategies to densify the graph by link predictions. Later, this work proposes a novel Electra module that pre-trains the LTR model as a discriminator module for node reconstruction tasks, where node features of selected edges would be randomly masked and reconstructed by a generator, and the discriminator learns to classify whether the reconstructed features are the original or replaced as well as perform correct ranking. Finally, the pre-trained discriminator module, rather than the generator, would be fine-tuned on the labeled graph. We carried out extensive offline and online evaluations using the real-world web traffic of Baidu search engine. The results show that RankElectra could significantly boost the ranking performance of Baidu Search compared with numbers of competitor systems.

Graph Neural Networks (GNNs) have become critical in various domains such as online advertising but face scalability challenges due to the growing size of graph data, leading to the needs for advanced distributed GPU computation strategies across multiple nodes. This paper presents PGLBox-Cluster, a robust distributed graph learning framework constructed atop the PaddlePaddle platform, implemented to efficiently process graphs comprising billions of nodes and edges. Through strategic partitioning of the model, node attributes, and graph data and leveraging industrial-grade RPC and NCCL for communication, PGLBox-Cluster facilitates effective distributed computation. The extensive experimental results confirm that PGLBox-Cluster achieves a 1.94x to 2.93x speedup over the single-node configuration, significantly elevating graph neural network scalability and efficiency by handling datasets exceeding 3 billion nodes and 120 billion edges with its novel asynchronous communication and graph partitioning techniques. The repository is released at This Link.

Drug discovery often relies on the successful prediction of protein-ligand binding affinity. Recent advances have shown great promise in applying graph neural networks (GNNs) for better affinity prediction by learning the representations of protein-ligand complexes. However, existing solutions usually treat protein-ligand complexes as topological graph data, thus the 3D geometry-based biomolecular structural information is not fully utilized. The essential intermolecular interactions with long-range dependencies, including type-wise interactions and molecule-wise interactions, are also neglected in GNN models. To this end, we propose a geometry-aware interactive graph neural network (GIaNt) which consists of two components: 3D geometric graph learning network (3DG-Net) and pairwise interactive learning network (Pi-Net). Specifically,3DG-Netiteratively performs the node-edge interaction process to update embeddings of nodes and edges in a unified framework while preserving the 3D geometric factors among atoms, including spatial distance, polar angle and dihedral angle information in 3D space. Moreover,Pi-Netis adopted to incorporate both element type-level and molecule-level interactions. Specially, interactive edges are gathered with a subsequent reconstruction loss to reflect the global type-level interactions. Meanwhile, a pairwise attentive pooling scheme is designed to identify the critical interactive atoms for complex representation learning from a semantic view. An exhaustive experimental study on two benchmarks verifies the superiority ofGIaNt.

While learning to rank (LTR) is widely employed in web searches to prioritize pertinent webpages from the retrieved contents based on input queries, traditional LTR models stumble over two principal stumbling blocks leading to subpar performance: 1) the lack of well-annotated query-webpage pairs with ranking scores to cover search queries of various popularity, debilitating their coverage of search queries across the popularity spectrum, and 2) ill-trained models that are incapable of inducing generalized representations for LTR, culminating in overfitting. To tackle the above challenges, we proposed a Generative Semi-Supervised Pre-trained (GS2P) Learning to Rank model. Specifically, GS2P first generates pseudo-labels for the unlabeled samples using tree-based LTR models after a series of co-training procedures, then learns the representations of query-webpage pairs with self-attentive transformers via both discriminative (LTR) and generative (denoising autoencoding for reconstruction) losses. Finally, GS2P boosts the performance of LTR through incorporating Random Fourier Features to over-parameterize the models into “interpolating regime”, so as to enjoy the further descent of generalization errors with learned representations. We conduct extensive offline experiments on a publicly available dataset and a real-world dataset collected from a large-scale search engine. The results show that GS2P can achieve the best performance on both datasets, compared to baselines. We also deploy GS2P at a large-scale web search engine with realistic traffic, where we can still observe significant improvement in real-world applications. GS2P performs consistently in both online and offline experiments.

Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning. Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism. Though one could lower the complexity of Transformers by inducing the sparsity in point-wise self-attentions for LTTF, the limited information utilization prohibits the model from exploring the complex dependencies comprehensively. To this end, we propose an efficient Transformer-based model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects: (i) an encoder-decoder architecture incorporating a linear complexity without sacrificing information utilization is proposed on top of sliding-window attention and Stationary and Instant Recurrent Network (SIRN); (ii) a module derived from the normalizing flow is devised to further improve the information utilization by inferring the outputs with the latent variables in SIRN directly; (iii) the inter-series correlation and temporal dynamics in time-series data are modeled explicitly to fuel the downstream self-attention mechanism. Extensive experiments on seven real-world datasets demonstrate that Conformer outperforms the state-of-the-art methods on LTTF and generates reliable prediction results with uncertainty quantification.

Training images with data transformations, e.g., crops, shifts, rotations and color distortions, have been suggested as contrastive examples to evaluate the robustness of deep neural networks against data noises [1]. In this work, we propose a practical framework ContRE (which is the meaning of “against” in French) that uses Contrastive examples for DNN Robustness Estimation. Specifically, ContRE follows the assumption in [2], [3] that robust DNN models with good generalization performance are capable of extracting a consistent set of features and making consistent predictions from the same image under varying data transformations. Incorporating with a set of randomized strategies for well-designed data transformations over the training set, ContREadopts classification errors and Fisher ratios on the generated contrastive examples to assess and analyze the robustness of DNN models, which correlates to the models’ generalization performance. To show the effectiveness and efficiency of ContRE, extensive experiments have been done using various DNN models, e.g., ResNet, VGGNet, DenseNet, EfficientNet, etc., on three open source benchmark datasets, i.e., CIFAR-10, CIFAR-100, and ImageNet, with thorough ablation studies and applicability analyses. Our experiment results confirm that ❨1❩ behaviors of deep models on contrastive examples are strongly correlated to what on the testing set, and ❨2❩ the robustness that ContRE calculates is a robust measure of generalization performance complementing to the testing set in various settings. Codes is to be publicly available.

Both Transformer and Graph Neural Networks (GNNs) have been used in learning to rank (LTR), they however adhere to two distinct-yet-complementary problem formulations, i.e., ranking score regression based on query-webpage pairs and link prediction within query-webpage bipartite graphs, respectively. Though it is possible to pre-train GNNs or Transformers on source datasets and fine-tune them subject to sparsely annotated LTR datasets separately, the source-target distribution shifts across the pairs and bipartite graphs domains make it extremely difficult to integrate these diverse models into a single LTR framework at a web-scale. We introduce the novel MPGraf model, which utilizes a modular and capsule-based pre-training approach, aiming to incorporate regression capacities from Transformers and link prediction capabilities of GNNs cohesively. Specifically, rather than simply combining these two modules in stacking or parallelizing architectures, MPGraf proposes a three-step method to (1) construct query-webpage graphs from sparsely annotated query-webpage pairs, (2) pre-train the Transformer and GNN modules using source datasets from two domains, and (3) “surgically” fine-tune the integrated models with pretrained weights while handling the distribution shifts of two domains. Extensive experiments have been done to evaluate the performance of MPGraf using real-world datasets collected from large-scale search engines. The results show that MPGraf can outperform baseline algorithms on several major metrics. Further, we deploy and evaluate MPGraf atop a large-scale search engine with realistic web traffics via A/B tests, where we can still observe significant improvement. MPGraf performs consistently in both offline and online evaluations.

While having been used widely for large-scale recommendation and online advertising, the Graph Neural Network (GNN) has demonstrated its representation learning capacity to extract embeddings of nodes and edges through passing, transforming, and aggregating information over the graph. In this work, we propose PGLBox1 - a multi-GPU graph learning framework based on PaddlePaddle [24], incorporating with optimized storage, computation, and communication strategies, to train deep GNNs based on web-scale graphs for the recommendation. Specifically, PGLBox adopts a hierarchical storage system with three layers to facilitate I/O, where graphs and embeddings are stored in the HBMs and SSDs, respectively, with MEMs as the cache. To fully utilize multi-GPUs and I/O bandwidth, PGLBox proposes an asynchronous pipeline with three stages - it first samples the subgraphs from the input graph, then pulls & updates embeddings and trains GNNs on the subgraph with parameters updating queued at the end of the pipeline. Thanks to the capacity of PGLBox in handling web-scale graphs, it becomes feasible to unify the view of GNN-based recommendation tasks for multiple advertising verticals and fuse all these graphs into a unified yet huge one. We evaluate PGLBox using a bucket of realistic GNN training tasks for the recommendation, and compare the performance of PGLBox on top of a multi-GPU server (Tesla A100×8) and the legacy training system based on a 40-node MPI cluster at Baidu. The overall comparisons show that PGLBox could save up to 55% monetary cost for training GNN models, and achieve up to 14× training speedup with the same accuracy as the legacy trainer. The open-source implementation of PGLBox is available at https://github.com/PaddlePaddle/PGL/tree/main/apps/PGLBox.

While Learning to Rank (LTR) models on top of transformers have been widely adopted to achieve decent performance, it is still challenging to train the model with sufficient data as only an extremely small number of query-webpage pairs could be annotated versus trillions of webpages available online and billions of web search queries everyday. In the meanwhile, industry research communities have released a number of open-source LTR datasets with well annotations but incorporating different designs of LTR features/labels (i.e., heterogeneous domains). In this work, inspired by the recent progress in pre-training transformers for performance advantages, we study the problem of pre-training LTR models using both labeled and unlabeled samples, especially we focus on the use of well-annotated samples in heterogeneous open-source LTR datasets to boost the performance of pre-training. Hereby, we propose S2phere-Semi-Supervised Pre-training with Heterogeneous LTR data strategies for LTR models using both unlabeled and labeled query-webpage pairs across heterogeneous LTR datasets. S2phere consists of a three-step approach: (1) Semi-supervised Feature Extraction Pre-training via Perturbed Contrastive Loss, (2) Cross-domain Ranker Pre-training over Heterogeneous LTR Datasets and (3) End-to-end LTR Fine-tuning via Modular Network Composition. Specifically, given an LTR model composed of a backbone (the feature extractor), a neck (the module to reason the orders) and a head (the predictor of ranking scores), S2phere uses unlabeled/labeled data from the search engine to pre-train the backbone in Step (1) via semi-supervised learning; then Step (2) incorporates multiple open-source heterogeneous LTR datasets to improve pre-training of the neck module as shared parameters of cross-domain learning; and finally, S2phere in Step (3) composes the backbone and neck with a randomly-initialized head into a whole LTR model and fine-tunes the model using search engine data with various learning strategies. Extensive experiments have been done with both offline experiments and online A/B Test on top of Baidu search engine. The comparisons against numbers of baseline algorithms confirmed the advantages of S2phere in producing high-performance LTR models for web-scale search.

Traffic congestion on a road segment typically begins as a small-scale spatiotemporal event that can then propagate throughout a road network and produce large-scale disruptions to a transportation system. In current techniques for the analysis of network flow, data is often aggregated to relatively large (e.g. 5 min) discrete time steps that obscure the small-scale spatiotemporal interactions that drive larger-scale dynamics. We propose a new method that handles fine-grained data to better capture those dynamics. Propagation patterns of traffic congestion are represented as spatiotemporally connected events. Each event is captured as a time series at the temporal resolution of the available trajectory data and at the spatial resolution of the network edge. The spatiotemporal propagation patterns of traffic congestion are captured using Dynamic Time Warping and represented as a set of directed acyclic graphs of spatiotemporal events. Results from this method are compared to an existing method using fine-grained data derived from an agent-based model of traffic simulation. Our method outperforms the existing method. Our method also successfully detects congestion propagation patterns that were reported by media news using sparse real-world data derived from taxis.

Predicting air quality in fine spatiotemporal granularity is of great importance for air pollution control and urban sustainability. However, existing studies are either focused on predicting station-wise future air quality, or inferring current air quality for unmonitored regions. How to accurately forecast future air quality for these unmonitored regions in a fine granularity remains an unexplored problem. In this paper, we propose the Self-Supervised Hierarchical Graph Neural Network (SSH-GNN), for fine-grained air quality forecasting in a semi-supervised way. Specifically, to augment spatially sparse air quality observations, SSH-GNN first approximates the city-wide air quality distribution based on historical readings and various urban contextual factors (e.g., weather conditions and traffic flows). Then, we propose a hierarchical recurrent graph neural network to make city-wide predictions, which encodes the spatial hierarchy of urban regions for long-range spatiotemporal correlation modeling. Moreover, by leveraging spatiotemporal self-supervision strategies, SSH-GNN exploits both universal topological and contextual patterns to further enhance the forecasting effectiveness. Extensive experiments on two real-world datasets show that SSH-GNN significantly outperforms the state-of-the-art algorithms.

Whilelearning to rank(LTR) has been widely used in web search to prioritize most relevant webpages among the retrieved contents subject to the input queries, the traditional LTR models fail to deliver decent performance due to two main reasons: 1) the lack of well-annotated query-webpage pairs with ranking scores to cover search queries of various popularity, and 2) ill-trained models based on a limited number of training samples with poor generalization performance. To improve the performance of LTR models, tremendous efforts have been done from above two aspects, such as enlarging training sets with pseudo-labels of ranking scores by self-training, or refining the features used for LTR through feature extraction and dimension reduction. Though LTR performance has been marginally increased, we still believe these methods could be further improved in the newly-fashioned “interpolating regime”. Specifically, instead of lowering the number of features used for LTR models, our work proposes to transform original data with random Fourier feature, so as to over-parameterize the downstream LTR models (e.g., GBRank or LightGBM) with features in ultra-high dimensionality and achieve superb generalization performance. Furthermore, rather than self-training with pseudo-labels produced by the same LTR model in a “self-tuned” fashion, the proposed method incorporates the diversity of prediction results between the listwise and pointwise LTR models while co-training both models with a cyclic labeling-prediction pipeline in a “ping-pong” manner. We deploy the proposedCo-trained andOver-parameterizedLTRsystemCOLTRat Baidu search and evaluateCOLTRwith a large number of baseline methods. The results show thatCOLTRcould achieve$\Delta NDCG_{4}$= 3.64%$\sim$4.92%, compared to baselines, under various ratios of labeled samples. We also conduct a 7-day A/B Test using the realistic web traffics of Baidu Search, where we can still observe significant performance improvement around$\Delta NDCG_{4}$= 0.17%$\sim$0.92% in real-world applications.COLTRperforms consistently both in online and offline experiments.

Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly improve and accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this article, we propose a novel regularized transfer learning framework \operatorname{DELTA} , namely DE ep L earning T ransfer using Feature Map with A ttention . Instead of constraining the weights of neural network, \operatorname{DELTA} aims at preserving the outer layer outputs of the source network. Specifically, in addition to minimizing the empirical loss, \operatorname{DELTA} aligns the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in a supervised learning manner. We evaluate \operatorname{DELTA} with the state-of-the-art algorithms, including L^2 and \emph {L}^2\text{-}SP . The experiment results show that our method outperforms these baselines with higher accuracy for new tasks. Code has been made publicly available. 1

Regularization that incorporates the linear combination of empirical loss and explicit regularization terms as the loss function has been frequently used for many machine learning tasks. The explicit regularization term is designed in different types, depending on its applications. While regularized learning often boost the performance with higher accuracy and faster convergence, the regularization would sometimes hurt the empirical loss minimization and lead to poor performance. To deal with such issues in this work, we propose a novel strategy, namely Gr adients O rthogonal D ecomposition ( GrOD ), that improves the training procedure of regularized deep learning. Instead of linearly combining gradients of the two terms, GrOD re-estimates a new direction for iteration that does not hurt the empirical loss minimization while preserving the regularization affects, through orthogonal decomposition. We have performed extensive experiments to use GrOD improving the commonly used algorithms of transfer learning [ 2 ], knowledge distillation [ 3 ], and adversarial learning [ 4 ]. The experiment results based on large datasets, including Caltech 256 [ 5 ], MIT indoor 67 [ 6 ], CIFAR-10 [ 7 ], and ImageNet [ 8 ], show significant improvement made by GrOD for all three algorithms in all cases.

While online advertising is one of the major sources of income for search engines, pumping up the incomes from business advertisements while ensuring the user experience becomes a challenging but emerging area. Designing high-quality advertisements with persuasive content has been proved as a way to increase revenues through improving the Click-Through Rate (CTR). However, it is difficult to scale up the design of high-quality ads, due to the lack of automation in creativity. In this paper, we present Commonsense-Enriched Advertisement on Search Engine (CHASE) --- a system for the automatic generation of persuasive ads. CHASE adopts a specially designed language model that fuses the keywords, commonsense-related texts, and marketing contents to generate persuasive advertisements. Specifically, the language model has been pre-trained using massive contents of explicit knowledge and fine-tuned with well-constructed quasi-parallel corpora with effective control of the proportion of commonsense in the generated ads and fitness to the ads' keywords. The effectiveness of the proposed method CHASE has been verified by real-world web traffics for search and manual evaluation. In A/B tests, the advertisements generated by CHASE would bring 11.13% CTR improvement. The proposed model has been deployed to cover three advertisement domains (which are kid education, psychological counseling, and beauty e-commerce) at Baidu, the world's largest Chinese search engine, with adding revenue of about 1 million RMB (Chinese Yuan) per day.

Drug discovery often relies on the successful prediction of protein-ligand binding affinity. Recent advances have shown great promise in applying graph neural networks (GNNs) for better affinity prediction by learning the representations of protein-ligand complexes. However, existing solutions usually treat protein-ligand complexes as topological graph data, thus the biomolecular structural information is not fully utilized. The essential long-range interactions among atoms are also neglected in GNN models. To this end, we propose a structure-aware interactive graph neural network (SIGN) which consists of two components: polar-inspired graph attention layers (PGAL) and pairwise interactive pooling (PiPool). Specifically, PGAL iteratively performs the node-edge aggregation process to update embeddings of nodes and edges while preserving the distance and angle information among atoms. Then, PiPool is adopted to gather interactive edges with a subsequent reconstruction loss to reflect the global interactions. Exhaustive experimental study on two benchmarks verifies the superiority of SIGN.

In modern internet industries, deep learning based recommender systems have became an indispensable building block for a wide spectrum of applications, such as search engine, news feed, and short video clips. However, it remains challenging to carry the well-trained deep models for online real-time inference serving, with respect to the time-varying web-scale traffics from billions of users, in a cost-effective manner. In this work, we present JIZHI - a Model-as-a-Service system - that per second handles hundreds of millions of online inference requests to huge deep models with more than trillions of sparse parameters, for over twenty real-time recommendation services at Baidu, Inc. In JIZHI, the inference workflow of every recommendation request is transformed to a Staged Event-Driven Pipeline (SEDP), where each node in the pipeline refers to a staged computation or I/O intensive task processor. With traffics of real-time inference requests arrived, each modularized processor can be run in a fully asynchronized way and managed separately. Besides, JIZHI introduces the heterogeneous and hierarchical storage to further accelerate the online inference process by reducing unnecessary computations and potential data access latency induced by ultra-sparse model parameters. Moreover, an intelligent resource manager has been deployed to maximize the throughput of JIZHI over the shared infrastructure by searching the optimal resource allocation plan from historical logs and fine-tuning the load shedding policies over intermediate system feedback. Extensive experiments have been done to demonstrate the advantages of JIZHI from the perspectives of end-to-end service latency, system-wide throughput, and resource consumption. Since launched in July 2019, JIZHI has helped Baidu saved more than ten million US dollars in hardware and utility costs per year while handling 200% more traffics without sacrificing the inference efficiency.

Ensemble learning is a widely used technique to train deep convolutional neural networks (CNNs) for improved robustness and accuracy. While existing algorithms usually first train multiple diversified networks and then assemble these networks as an aggregated classifier, we propose a novel learning paradigm, namely, “In-Network Ensemble” ( INE ) that incorporates the diversity of multiple models through training a SINGLE deep neural network. Specifically, INE segments the outputs of the CNN into multiple independent classifiers, where each classifier is further fine-tuned with better accuracy through a so-called diversified knowledge distillation process . We then aggregate the fine-tuned independent classifiers using an Averaging-and-Softmax operator to obtain the final ensemble classifier. Note that, in the supervised learning settings, INE starts the CNN training from random, while, under the transfer learning settings, it also could start with a pre-trained model to incorporate the knowledge learned from additional datasets. Extensive experiments have been done using eight large-scale real-world datasets, including CIFAR, ImageNet, and Stanford Cars, among others, as well as common deep network architectures such as VGG, ResNet, and Wide ResNet. We have evaluated the method under two tasks: supervised learning and transfer learning. The results show that INE outperforms the state-of-the-art algorithms for deep ensemble learning with improved accuracy.

Stochastic Gradient Langevin Dynamics (SGLD) have been widely used for Bayesian sampling from certain probability distributions, incorporating derivatives of the log-posterior. With the derivative evaluation of the log-posterior distribution, SGLD methods generate samples from the distribution through performing as a thermostats dynamics that traverses over gradient flows of the log-posterior with certainly controllable perturbation. Even when the density is not known, existing solutions still can first learn the kernel density models from the given datasets, then produce new samples using the SGLD over the kernel density derivatives. In this work, instead of exploring new samples from kernel spaces, a novel SGLD sampler, namely, Randomized Measurement Langevin Dynamics (RMLD) is proposed to sample the high-dimensional sparse representations from the spectral domain of a given dataset. Specifically, given a random measurement matrix for sparse coding, RMLD first derives a novel likelihood evaluator of the probability distribution from the loss function of LASSO, then samples from the high-dimensional distribution using stochastic Langevin dynamics with derivatives of the logarithm likelihood and Metropolis–Hastings sampling. In addition, new samples in low-dimensional measuring spaces can be regenerated using the sampled high-dimensional vectors and the measurement matrix. The algorithm analysis shows that RMLD indeed projects a given dataset into a high-dimensional Gaussian distribution with Laplacian prior, then draw new sparse representation from the dataset through performing SGLD over the distribution. Extensive experiments have been conducted to evaluate the proposed algorithm using real-world datasets. The performance comparisons on three real-world applications demonstrate the superior performance of RMLD beyond baseline methods.

As it is difficult to directly share data among different organizations, data federation brings new opportunities to the data-related cooperation among different organizations by providing abstract data interfaces. With the development of Cloud computing, organizations store data on the Cloud to achieve elasticity and scalability for data processing. The existing data placement approaches generally only consider one aspect, which is either communication cost or time cost, and do not consider the features of jobs that process the data. In this paper, we propose an approach to enable secure data processing on the Cloud with the data from different organizations. The approach consists of a data federation platform for secure data processing on the Cloud named FedCube and a greedy data placement algorithm that creates a plan to store data on the Cloud in order to achieve multiple objectives based on a cost model. The cost model is composed of two objectives, i.e., reducing both monetary cost and execution time. We present an experimental evaluation by comparing our data placement algorithm with the existing methods based on the data federation platform. The experiments show that our proposed algorithm significantly reduce the total cost (up to 69.8%).

As the recent COVID-19 outbreak rapidly expands all over the world, various containment measures have been carried out to fight against the COVID-19 pandemic. In Mainland China, the containment measures consist of three types, i.e., Wuhan travel ban, intra-city quarantine and isolation, and intercity travel restriction. In order to carry out the measures, local economy and information acquisition play an important role. In this paper, we investigate the correlation of local economy and the information acquisition on the execution of containment measures to fight against the COVID-19 pandemic in Mainland China. First, we use a parsimonious model, i.e., SIR-X model to estimate the parameters, which represent the execution of intra-city quarantine and isolation in major cities of Mainland China. In order to understand the execution of intra-city quarantine and isolation, we analyze the correlation between the representative parameters including local economy, mobility, and information acquisition. To this end, we collect the data of Gross Domestic Product (GDP), the inflows from Wuhan and outflows, and the COVID-19 related search frequency from a widely-used Web mapping service, i.e., Baidu Maps, and Web search engine, i.e., Baidu Search Engine, in Mainland China. Based on the analysis, we confirm the strong correlation between the local economy and the execution of information acquisition in major cities of Mainland China. We further evidence that, although the cities with high GDP per capita attract more inflows from Wuhan, people are more likely to conduct the quarantine measure and to reduce travelling to other cities. Finally, the correlation analysis using search data shows that well-informed individuals are likely to carry out containment measures.

Sparse Discriminant Analysis (SDA) has been widely used to improve the performance of classical Fisher’s Linear Discriminant Analysis in supervised metric learning, feature selection, and classification. With the increasing needs of distributed data collection, storage, and processing, enabling the Sparse Discriminant Learning to embrace the multi-party distributed computing environments becomes an emerging research topic. This article proposes a novel multi-party SDA algorithm, which can learn SDA models effectively without sharing any raw data and basic statistics among machines. The proposed algorithm (1) leverages the direct estimation of SDA to derive a distributed loss function for the discriminant learning, (2) parameterizes the distributed loss function with local/global estimates through bootstrapping, and (3) approximates a global estimation of linear discriminant projection vector by optimizing the “distributed bootstrapping loss function” with gossip-based stochastic gradient descent. Experimental results on both synthetic and real-world benchmark datasets show that our algorithm can compete with the aggregated SDA with similar performance, and significantly outperforms the most recent distributed SDA in terms of accuracy and F1-score.

Federated machine learning systems have been widely used to facilitate the joint data analytics across the distributed datasets owned by the different parties that do not trust each others. In this paper, we proposed a novel Gradient Boosting Machines (GBM) framework SecureGBM built-up with a multi-party computation model based on semi-homomorphic encryption, where every involved party can jointly obtain a shared Gradient Boosting machines model while protecting their own data from the potential privacy leakage and inferential identification. More specific, our work focused on a specific “dualparty” secure learning scenario based on two parties — both party own an unique view (i.e., attributes or features) to the sample group of samples while only one party owns the labels. In such scenario, feature and label data are not allowed to share with others.To achieve the above goal, we firstly extent — LightGBM — a well known implementation of tree-based GBM through covering its key operations for training and inference with SEAL homomorphic encryption schemes. However, the performance of such re-implementation is significantly bottle-necked by the explosive inflation of the communication payloads, based on ciphertexts subject to the increasing length of plaintexts. In this way, we then proposed to use stochastic approximation techniques to reduced the communication payloads while accelerating the overall training procedure in a statistical manner. Our experiments using the real-world data showed that SecureGBM can well secure the communication and computation of LightGBM training and inference procedures for the both parties while only losing less than 3% AUC, using the same number of iterations for gradient boosting, on a wide range of benchmark datasets. More specific, compared to LightGBM, the proposed SecureGBM would slowdown with $3\mathrm{x} \sim 64\mathrm{x}$ time consumption per iteration in the training procedure, while SecureGBM becomes more and more efficient when the scale of the training dataset increases (i.e., the larger training set, the lower slowdown ratio).

Transfer learning have been frequently used to improve deep neural network training through incorporating weights of pre-trained networks as the starting-point of optimization for regularization. While deep transfer learning can usually boost the performance with better accuracy and faster convergence, transferring weights from inappropriate networks hurts training procedure and may lead to even lower accuracy. In this paper, we consider deep transfer learning as minimizing a linear combination of empirical loss and regularizer based on pre-trained weights, where the regularizer would restrict the training procedure from lowering the empirical loss, with conflicted descent directions (e.g., derivatives). Following the view, we propose a novel strategy making regularization-based Deep Transfer learning Never Hurt (DTNH) that, for each iteration of training procedure, computes the derivatives of the two terms separately, then re-estimates a new descent direction that does not hurt the empirical loss minimization while preserving the regularization affects from the pre-trained weights. Extensive experiments have been done using common transfer learning regularizers, such as L2-SP and knowledge distillation, on top of a wide range of deep transfer learning benchmarks including Caltech, MIT indoor 67, CIFAR-10 and ImageNet. The empirical results show that the proposed descent direction estimation strategy DTNH can always improve the performance of deep transfer learning tasks based on all above regularizers, even when transferring pre-trained weights from inappropriate networks. All in all, DTNH strategy can improve state-of-the-art regularizers in all cases with 0.1%-7% higher accuracy in all experiments.

Finding a right doctor with suitable expertise that meets one's health needs is important yet challenging. In this paper, we study the problem of finding high-rated doctors for a specific disease using imbalanced and heterogeneous healthcare experience rating data. We develop a data analytical framework, namely Dr. Right!, which incorporates the so-called network-textual embeddings, together with data-imbalance-aware mixture multi-classification models to rate doctors per specific disease. First, Dr. Right! collects the comments and rating records from patients for doctors on specific diseases from an online hospital and constructs a doctor-patient-disease network, where every edge weight is a pairwise average rating (experience score) among doctors, patients, and diseases. Then, Dr. Right! learns the embeddings of patient experiences from textual comments using the Word2Vec, as well as the embeddings of doctors and diseases from the doctor-patient-disease network via the Node2Vec. The two types of embeddings are fused to represent a doctor-patient pair. With the embedding representations of doctor-patient pairs, Dr. Right! learns an adaptively-weighted mixture multi-classification model to map a doctor-disease pair to an experience rating score, while addressing the challenges of data imbalance and group heterogeneity. Finally, extensive experimental results demonstrate the enhanced performances of Dr. Right! for predicting the disease-specific experience scores of doctors.

Data quality and budget are two primary concerns in urban-scale mobile crowdsensing. Traditional research on mobile crowdsensing mainly takes sensing coverage ratio as the data quality metric rather than the overall sensed data error in the target-sensing area. In this article, we propose to leverage spatiotemporal correlations among the sensed data in the target-sensing area to significantly reduce the number of sensing task assignments. In particular, we exploit both intradata correlations within the same type of sensed data and interdata correlations among different types of sensed data in the sensing task. We propose a novel crowdsensing task allocation framework called SPACE-TA (SPArse Cost-Effective Task Allocation) , combining compressive sensing, statistical analysis, active learning, and transfer learning, to dynamically select a small set of subareas for sensing in each timeslot (cycle), while inferring the data of unsensed subareas under a probabilistic data quality guarantee. Evaluations on real-life temperature, humidity, air quality, and traffic monitoring datasets verify the effectiveness of SPACE-TA. In the temperature-monitoring task leveraging intradata correlations, SPACE-TA requires data from only 15.5% of the subareas while keeping the inference error below 0.25°C in 95% of the cycles, reducing the number of sensed subareas by 18.0% to 26.5% compared to baselines. When multiple tasks run simultaneously, for example, for temperature and humidity monitoring, SPACE-TA can further reduce ∼10% of the sensed subareas by exploiting interdata correlations.

Sparse Discriminant Analysis (SDA) has been widely used to improve the performance of classical Fisher's Linear Discriminant Analysis in supervised metric learning, feature selection and classification. With the increasing needs of distributed data collection, storage and processing, enabling the Sparse Discriminant Learning to embrace the Multi-Party distributed computing environments becomes an emerging research topic. This paper proposes a novel Multi-Party SDA algorithm, which can learn SDA models effectively without sharing any raw dataand basic statistics among machines. The proposed algorithm 1) leverages the direct estimation of SDA [1] to derive a distributed loss function for the discriminant learning, 2) parameterizes the distributed loss function with local/global estimates through bootstrapping, and 3) approximates a global estimation of linear discriminant projection vector by optimizing the "distributed bootstrapping loss function" with gossip-based stochastic gradient descent. Experimental results on both synthetic and real-world benchmark datasets show that our algorithm can compete with the centralized SDA with similar performance, and significantly outperforms the most recent distributed SDA [2] in terms of accuracy and F1-score.

Linear Discriminant Analysis (LDA) is widely-used for supervised dimension reduction and linear classification. Classical LDA, however, suffers from the ill-posed estimation problem on data with high dimension and low sample size (HDLSS). To cope with this problem, in this paper, we propose an Adaptive Wishart Discriminant Analysis (AWDA) for classification, that makes predictions in an ensemble way. Comparing to existing approaches, AWDA has two advantages: 1) leveraging theWishart distribution, AWDA ensembles multiple LDA classifiers parameterized by the sampled covariance matrices via a Bayesian Voting Scheme, which theoretically improves the robustness of classification, compared to LDA classifiers using a single (probably ill-posed) covariance matrix estimator; 2) AWDA updates the weights for voting optimally to adapt the local information of each new input data, so as to enable the nonlinear classification. Theoretical analysis indicates that AWDA guarantees a close approximation to the optimal Bayesian inference and thus achieves robust performance on high dimensional data. Extensive experiments on real-world datasets show that our approach outperforms state-of-the-art algorithms by a large margin.

Electronic health records (EHR) provide a rich source of temporal data that present a unique opportunity to characterize disease patterns and risk of imminent disease. While many data-mining tools have been adopted for EHR-based disease early detection, linear discriminant analysis (LDA) is one of the most commonly used statistical methods. However, it is difficult to train an accurate LDA model for early disease diagnosis when too few patients are known to have the target disease. Furthermore, EHR data are heterogeneous with significant noise. In such cases, the covariance matrices used in LDA are usually singular and estimated with a large variance. This article presents Daehr , an extension of the LDA framework using electronic health record data to address these issues. Beyond existing LDA analyzers, we propose Daehr to (1) eliminate the data noise caused by the manual encoding of EHR data and (2) lower the variance of parameter (covariance matrices) estimation for LDA models when only a few patients’ EHR are available for training. To achieve these two goals, we designed an iterative algorithm to improve the covariance matrix estimation with embedded data-noise/parameter-variance reduction for LDA. We evaluated Daehr extensively using the College Health Surveillance Network, a large, real-world EHR dataset. Specifically, our experiments compared the performance of LDA to three baselines (i.e., LDA and its derivatives) in identifying college students at high risk for mental health disorders from 23 U.S. universities. Experimental results demonstrate Daehr significantly outperforms the three baselines by achieving 1.4%--19.4% higher accuracy and a 7.5%--43.5% higher F1-score.

According to a 2014 Spring American College Health Association Survey, almost 50% of college students reported feeling things were hopeless and that it was difficult to function within the last 12 months. More than 80% reported feeling overwhelmed and exhausted by their responsibilities. This critical subpopulation of Americans is facing significant levels of mental health disorders, challenging colleges to provide accessible and high quality behavioral health care. However, psychiatric disorders are frequently unrecognized in primary care settings, posing physical, emotional, economic, and social burdens to patients and others. Towards the goal of earlier identification and treatment of mental health disorders, this paper proposes M-SEQ, an early detection framework for anxiety/depression using electronic health data from primary care visit sequences. Specifically, compared to existing methods that predict a future disease state using frequency of diagnoses in a patient's medical history, we hypothesize that future disease might also be correlated with the temporal orders of diagnoses. Thus, M-SEQ first discovers a set of diagnosis codes that are discriminative of anxiety/depression, and then extracts each diagnosis pair from each patient's health record to represent the temporal orders of diagnoses. Further, it incorporates the extracted temporal order information with the existing representation to predict whether a patient is at risk of anxiety/depression. We evaluate M-SEQ using the electronic health record (EHR) data of 213,112 college students from 10 schools participating in the College Health Surveillance Network (CHSN) from January 1, 2011 through December 31, 2014. The experimental results shows that our framework can detect a future diagnosis of anxiety and depression based on the primary care visit data up to 3 months in advance, with approximately 1%-4.5% higher accuracy, compared to baseline methods using frequency of diagnoses.

Mobile Crowdsensing (MCS) requires users to be motivated to participate. However, concerns regarding energy consumption and privacy—among other things—may compromise their willingness to join such a crowd. Our preliminary observations and analysis of common MCS applications have shown that the data transfer in MCS applications may incur significant energy consumption due to the 3G connection setup. However, if data are transferred in parallel with a traditional phone call, then such transfer can be done almost “for free”: with only an insignificant additional amount of energy required to piggy-back the data—usually incoming task assignments and outgoing sensor results—on top of the call. Here, we present an Energy-Efficient Mobile Crowdsensing (EEMC) framework where task assignments and sensing results are transferred in parallel with phone calls. The main objective, and the principal contribution of this article, is an MCS task assignment scheme that guarantees that a minimum number of anonymous participants return sensor results within a specified time frame, while also minimizing the waste of energy due to redundant task assignments and considering privacy concerns of participants. Evaluations with a large-scale real-world phone call dataset show that our proposed EEMC framework outperforms the baseline approaches, and it can reduce overall energy consumption in data transfer by 54--66% when compared to the 3G-based solution.