Pre-trained language models (PLMs) have been successfully used to build high-performance ranking models for large-scale information retrieval systems. However, traditional PLM-based ranking approaches face two key challenges: (1) these models use both sparse and dense content (such as the query/title and content of documents) as inputs, which may require different attention allocations; and (2) traditional PLM-based ranking approaches have identified multiple objectives to gauge user satisfaction with ranking results, but integrating these objectives into the end-to-end training process and the subsequent feature updates and iterations usually involves significant computational resource overhead. In this paper, we propose a novel PLM-based ranking approach M2oE Rank, Multi-objective Mixture-of-Experts (MoE) enhanced Ranking. Specifically, M2oERank lever-ages a context-aware PLM-based hierarchical encoder to extract semantic relevance between the query and the document title and content, while allowing for separate dense and sparse attention for different inputs. With the extracted semantic relevance repre-sentations, multifacet user satisfaction features and task-specific annotations, M2oERank employs an MoE module to perform multi-objective pre-training of ranking models focused on user satisfaction. Finally, M2oERank uses a weight fusion module that fuses outputs from the above experts to predict ranking scores. Moreover, we present a three-stage offline training strategy and the online system workflow for deploying M2oERank at web-scale search. To demonstrate the effectiveness of our proposed approach, we conduct extensive offline and online evaluations using real-world web traffic from Baidu Search. The comparisons against numbers of advanced baselines confirmed the advantages of M2oERank in producing high-performance ranking models for web-scale search.

Extracting query-document relevance from the sparse, biased clickthrough log is among the most fundamental tasks in the web search system. Prior art mainly learns a relevance judgment model with semantic features of the query and document and ignores directly counterfactual relevance evaluation from the clicking log. Though the learned semantic matching models can provide relevance signals for tail queries as long as the semantic feature is available. However, such a paradigm lacks the capability to introspectively adjust the biased relevance estimation whenever it conflicts with massive implicit user feedback. The counterfactual evaluation methods, on the contrary, ensure unbiased relevance estimation with sufficient click information. However, they suffer from the sparse or even missing clicks caused by the long-tailed query distribution.

Multivariate Time Series Classification (MTSC) is a prevalent but challenging problem in data mining. With the development of Deep Neural Networks (DNN), hundreds of deep models for MTSC have been proposed. However, most prior works only explicitly model the relationship between time series and classes and ignore the diversity of the relationship, suffering from insufficient information exploitation. In this paper, we propose a novel module named Multi-Relationship Modeling Module(MRM2) for more effective MTSC. MRM2 uses the classified labels to explicitly model not only the relationship between time series and classes, but also the relationship among time series, enabling the backbone to generate distinguishable embeddings. In addition, MRM2 is versatile because it can be combined with the existing backbones of DNN for end-to-end training. Finally, we conduct a series of ablation studies and comparative experiments on the real multivariate time series archive UEA. Experimental results indicate that MRM2 can significantly improve classification performance in most cases. Codes are available on GitHub1.

Memristor-based crossbars, which can achieve 1-2 orders of magnitude energy efficiency improvement over digital machines, have been introduced to accelerate the neural networks of machine learning tasks. Due to the high voltage pulses repeatedly applied onto memristors during programming and online tuning, the effective resistance ranges of the memristors actually decrease as a result of aging, which eventually impair the inference accuracy of the neural network running on the memristor-based crossbar. In this paper, we propose an algorithm-hardware co-design framework combining aging aware retraining and gradient sparsification to mitigate the impact of aging and extend the lifetime of the crossbar. Experimental results show that the proposed method can effectively increase the inference accuracy by up to 16% even with severe aging, while the crossbar lifetime can be extended by up to $2.7\times$.

The unbiased learning to rank (ULTR) problem has been greatly advanced by recent deep learning techniques and well-designed debias algorithms. However, promising results on the existing benchmark datasets may not be extended to the practical scenario due to some limitations of existing datasets. First, their semantic feature extractions are outdated while state-of-the-art large-scale pre-trained language models like BERT cannot be utilized due to the lack of original text. Second, display features are incomplete; thus in-depth study on ULTR is impossible such as the displayed abstract for analyzing the click necessary bias. Third, synthetic user feedback has been adopted by most existing datasets and real-world user feedback is greatly missing. To overcome these disadvantages, we introduce the Baidu-ULTR dataset. It involves randomly sampled 1.2 billion searching sessions and 7,008 expert annotated queries(397,572 query document pairs). Baidu-ULTR is the first billion-level dataset for ULTR. Particularly, it offers: (1)the original semantic features and pre-trained language models of different sizes; (2)sufficient display information such as position, displayed height, and displayed abstract, enabling the comprehensive study of multiple displayed biases; and (3)rich user feedback on search result pages (SERPs) like dwelling time, allowing for user engagement optimization and promoting the exploration of multi-task learning in ULTR. Furthermore, we present the design principle of Baidu-ULTR and the performance of representative ULTR algorithms on Baidu-ULTR. The Baidu-ULTR dataset and corresponding baseline implementations are available at https://github.com/ChuXiaokai/baiduultrdataset. The dataset homepage is available at https://searchscience.baidu.com/dataset.html.

Deep pre-trained language models (e.g., BERT) lead to remarkable headway in many Natural Language Processing tasks. Their superior capacity in perceiving textual data is also witnessed in semantic matching tasks (e.g., question answering, web search). Particularly for matching a pair of query and text candidate, the current state-of-the-arts usually rely on the semantic representations produced by BERT, and compute relevance scores with various interaction (i.e., matching) methods. However, they may 1) miss fine-grained phrase-level interaction between the input query and candidate context or 2) lack a thoughtful consideration of both effectiveness and efficiency. Motivated by this, we propose \hyttInteractor, a BERT-based semantic matching model with a flexible contextualized interaction paradigm. It is capable of capturing fine-grained phrase-level information in the interaction, and thus is more effective to be applied for semantic matching tasks. Moreover, we further facilitate \hyttInteractor with a novel partial attention scheme, which significantly reduces the computational cost while maintaining the high effectiveness. We conduct comprehensive experimental evaluations on three datasets. The results show that \hyttInteractor achieves superior effectiveness and efficiency for semantic matching.

Session-based recommendation has recently attracted increasing attention from both industry and academic communities. Previous models mostly focus on designing different models to fit the observed data, which can be quite sparse in real-world scenarios. To alleviate this problem, in this paper, we propose a novel generative session-based recommendation framework. The main building block of our idea is to develop a generator to simulate user sequential behaviors, which are leveraged to train and improve the target sequential recommender model. In order to generate high quality samples, we consider two aspects: (1) the rationality as a sequence of user behaviors, and (2) the informativeness for training the target model. To satisfy these requirements, we design a doubly adversarial network. The first adversarial module aims to make the generated samples conform to the underlying patterns of the real user sequential preference (rationality requirement). The second adversarial module is targeted at widening the model experiences by generating samples which can induce larger model losses (informativeness requirement). In our model, the samples are generated based on a reinforcement learning strategy, where the reward is related with both of the above aspects. In order to stable the training process, we introduce a self-paced regularizer to learn the agent in an easy-to-hard manner. We conduct extensive experiments based on real-world datasets to demonstrate the effectiveness of our model.

Incorporating review information into the recommender system has been demonstrated to be an effective method for boosting the recommendation performance. Previous research mainly focus on designing advanced architectures to better profile the users and items. However, the review information in realities can be highly sparse and imbalanced, which poses great challenges for effective user/item representations and satisfied performance enhancement. To alleviate this problem, in this paper, we propose to improve review-based recommendation by counterfactually augmenting the training samples. We focus on a common setting --- feature-aware recommendation, and the main building block of our idea lies in the counterfactual question: "what would be the user's decision if her feature-level preference had been different?''. When augmenting the training samples, we actively change the user preference (also called intervention), and predict the user feedback on the items based on pre-trained recommender models. Instead of changing the user preference in a random manner, we design a learning-based method to discover the samples which are more effective for model optimization. In order to improve the sample qualities, we propose two strategies --- constrained feature perturbation and frequency-based sampling --- to equip our model. Since the sample generation model can be not perfect, we theoretically analyze the relation between the model prediction error and the number of generated samples. As a byproduct, our framework can explain the user pair-wise preference, which is complementary to the traditional point-wise explanations. Extensive experiments demonstrate that our model can significantly improve the performance of the state-of-the-art methods.

Recently, significant progress has been made in time series classification with deep learning. However, using deep learning models to solve time series classification generally suffers from expensive calculations and difficulty of data labeling. In this work, we study self-supervised time series pre-training to overcome these challenges. Compared with the existing works, we focus on the universal and unlabeled time series pretraining. To this end, we propose a novel end-to-end neural network architecture based on self-attention, which is suitable for capturing long-term dependencies and extracting features from different time series. Then, we propose two different self-supervised pretext tasks for time series data type: Denoising and Similarity Discrimination based on DTW (Dynamic Time Warping). Finally, we carry out extensive experiments on 85 time series datasets (also known as UCR2015 [2]). Empirical results show that the time series model augmented with our proposed self-supervised pretext tasks achieves state-of-the-art / highly competitive results.

Post-click conversion, as a strong signal indicating the user preference, is salutary for building recommender systems. However, accurately estimating the post-click conversion rate (CVR) is challenging due to the selection bias, i.e., the observed clicked events usually happen on users' preferred items. Currently, most existing methods utilize counterfactual learning to debias recommender systems. Among them, the doubly robust (DR) estimator has achieved competitive performance by combining the error imputation based (EIB) estimator and the inverse propensity score (IPS) estimator in a doubly robust way. However, inaccurate error imputation may result in its higher variance than the IPS estimator. Worse still, existing methods typically use simple model-agnostic methods to estimate the imputation error, which are not sufficient to approximate the dynamically changing model-correlated target (i.e., the gradient direction of the prediction model). To solve these problems, we first derive the bias and variance of the DR estimator. Based on it, a more robust doubly robust (MRDR) estimator has been proposed to further reduce its variance while retaining its double robustness. Moreover, we propose a novel double learning approach for the MRDR estimator, which can convert the error imputation into the general CVR estimation. Besides, we empirically verify that the proposed learning scheme can further eliminate the high variance problem of the imputation learning. To evaluate its effectiveness, extensive experiments are conducted on a semi-synthetic dataset and two real-world datasets. The results demonstrate the superiority of the proposed approach over the state-of-the-art methods. The code is available at https://github.com/guosyjlu/MRDR-DL.

In this paper, we propose a new maximum entropy clustering algorithm by modified grey wolf optimizer (GWO), which modify the traditional GWO from twofold: First, nonlinear decay factor is constructed, which leads to flexible refined search; Second, proportional weights, which render the positions of distinct grey wolves, are adjusted adaptively according to the social hierarchy of them. Based on these two modifications, the rectified GWO could greatly improve both the accuracy and convergence rate. Experimental results on 12 benchmark functions demonstrate the effectiveness of those modifications. Furthermore, we utilize modified GWO to maximum entropy based fuzzy clustering problems. Experiments on 5 real datasets indicate the high performance and efficiency of the proposed approach.

Image captioning (IC) is a commonly-used technique for generating textual image description, which finds its applications on semantic image retrieval and multi-modal image understanding, among many others. This paper focuses on an important IC method specialized for generating aesthetic descriptions of images, i.e., aesthetic image captioning (AIC). Despite some effectiveness of initial work on AIC, their performances are inherently limited due to a lack of consideration of user preferences on aesthetics and better aesthetic feature, making it unusable for real-world applications where human users present a large variation on evaluating visual aesthetics of images. To tackle this, we propose a novel personalized aesthetic image caption (PAIC) approach for capturing and incorporating user preferences for AIC tasks. Our approach mainly contains Aesthetic feature Extraction Network(AEN), User Encoder network(UEN) and a personalized image caption model. AEN is designed to extract more expressive feature, UEN is introduced for learning the user vector from the limited information in our AVA-PCap dataset. Personalized image caption model is constructed to generate the caption when given the user id and photo pairs. The experimental results show that our methods outperform baselines by 10% , which is encouraging for a first step towards personalized aesthetic image caption.

Incorporating temporal information into recommender systems has recently attracted increasing attention from both the industrial and academic research communities. Existing methods mostly reduce the temporal information of behaviors to behavior sequences for subsequently RNN-based modeling. In such a simple manner, crucial time-related signals have been largely neglected. This paper aims to systematically investigate the effects of the temporal information in sequential recommendations. In particular, we firstly discover two elementary temporal patterns of user behaviors: "absolute time patterns'' and "relative time patterns'', where the former highlights user time-sensitive behaviors, e.g., people may frequently interact with specific products at certain time point, and the latter indicates how time interval influences the relationship between two actions. For seamlessly incorporating these information into a unified model, we devise a neural architecture that jointly learns those temporal patterns to model user dynamic preferences. Extensive experiments on real-world datasets demonstrate the superiority of our model, comparing with the state-of-the-arts.