While representation learning has been used to boost the performance of Learning-to-Rank (LTR) models through distilling key features for webpage ranking, the weak supervision signals extracted from users' sparse click-through data lead to inadequate representation of query-webpage pairs for ranking score prediction. Recent studies in generative LTR pre-training demonstrate the feasibility of incorporating reconstruction loss for enhanced ranking score prediction. However, LTR is afterall a regression task and it might be reasonable to find an alternate route that pre-trains LTR models with discriminative losses. Following the success of Electra in representation learning for natural language processing (NLP), this work proposes RankElectra that pre-trains the LTR model as a discriminator module inside a generative learning framework. Specifically, RankElectra first structures sparsely-annotated query-webpage pairs into a bipartite graph, with query and webpage feature vectors as node types and ranking scores as the connecting edges, and then leverages positive and negative extension strategies to densify the graph by link predictions. Later, this work proposes a novel Electra module that pre-trains the LTR model as a discriminator module for node reconstruction tasks, where node features of selected edges would be randomly masked and reconstructed by a generator, and the discriminator learns to classify whether the reconstructed features are the original or replaced as well as perform correct ranking. Finally, the pre-trained discriminator module, rather than the generator, would be fine-tuned on the labeled graph. We carried out extensive offline and online evaluations using the real-world web traffic of Baidu search engine. The results show that RankElectra could significantly boost the ranking performance of Baidu Search compared with numbers of competitor systems.

Training images with data transformations, e.g., crops, shifts, rotations and color distortions, have been suggested as contrastive examples to evaluate the robustness of deep neural networks against data noises [1]. In this work, we propose a practical framework ContRE (which is the meaning of “against” in French) that uses Contrastive examples for DNN Robustness Estimation. Specifically, ContRE follows the assumption in [2], [3] that robust DNN models with good generalization performance are capable of extracting a consistent set of features and making consistent predictions from the same image under varying data transformations. Incorporating with a set of randomized strategies for well-designed data transformations over the training set, ContREadopts classification errors and Fisher ratios on the generated contrastive examples to assess and analyze the robustness of DNN models, which correlates to the models’ generalization performance. To show the effectiveness and efficiency of ContRE, extensive experiments have been done using various DNN models, e.g., ResNet, VGGNet, DenseNet, EfficientNet, etc., on three open source benchmark datasets, i.e., CIFAR-10, CIFAR-100, and ImageNet, with thorough ablation studies and applicability analyses. Our experiment results confirm that ❨1❩ behaviors of deep models on contrastive examples are strongly correlated to what on the testing set, and ❨2❩ the robustness that ContRE calculates is a robust measure of generalization performance complementing to the testing set in various settings. Codes is to be publicly available.