Text relevance or text matching of query and product is an essential technique for the e-commerce search system to ensure that the displayed products can match the intent of the query. Many studies focus on improving the performance of the relevance model in search system. Recently, pre-trained language models like BERT have achieved promising performance on the text relevance task. While these models perform well on the offline test dataset, there are still obstacles to deploy the pre-trained language model to the online system as their high latency. The two-tower model is extensively employed in industrial scenarios, owing to its ability to harmonize performance with computational efficiency. Regrettably, such models present an opaque ''black box'' nature, which prevents developers from making special optimizations. In this paper, we raise deep Bag-o f-Words (DeepBoW) model, an efficient and interpretable relevance architecture for Chinese e-commerce. Our approach proposes to encode the query and the product into the sparse BoW representation, which is a set of word-weight pairs. The weight means the important or the relevant score between the corresponding word and the raw text. The relevance score is measured by the accumulation of the matched word between the sparse BoW representation of the query and the product. Compared to popular dense distributed representation that usually suffers from the drawback of black-box, the most advantage of the proposed representation model is highly explainable and interventionable, which is a superior advantage to the deployment and operation of online search engines. Moreover, the online efficiency of the proposed model is even better than the most efficient inner product form of dense representation. The proposed model is experimented on three different datasets for learning the sparse BoW representations, including the human-annotation set, the search-log set and the click-through set. Then the models are evaluated by experienced human annotators. Both the auto metrics and the online evaluations show our DeepBoW model achieves competitive performance while the online inference is much more efficient than the other models. Our DeepBoW model has already deployed to the biggest Chinese e-commerce search engine Taobao and served the entire search traffic for over 6 months.

To enhance the effectiveness of matching user requests with millions of online products, practitioners invest significant efforts in developing semantic relevance models on large-scale e-commerce platforms. Generally, such semantic relevance models are formulated as text-matching approaches, which measure the relevance between users' search queries and the titles of candidate items (i.e., products). However, we argue that conventional relevance methods may lead to sub-optimal performance due to the limited information provided by the titles of candidate items. To alleviate this issue, we suggest incorporating additional information about candidate items from multiple aspects, including their attributes and images. This could supplement the information that may not be fully provided by titles alone. To this end, we propose a multi-aspect semantic relevance model that takes into account the match between search queries and the title, attribute and image information of items simultaneously. The model is further enhanced through pre-training using several well-designed self-supervised and weakly-supervised tasks. Furthermore, the proposed model is fine-tuned using annotated data and distilled into a representation-based architecture for efficient online deployment. Experimental results show the proposed approach significantly improves relevance and leads to considerable enhancements in business metrics.

Text relevance or text matching of query and product is an essential technique for e-commerce search engine, which helps users find the desirable products and is also crucial to ensuring user experience. A major difficulty for e-commerce text relevance is the severe vocabulary gap between query and product. Recently, neural networks have been the mainstream for the text matching task owing to the better performance for semantic matching. Practical e-commerce relevance models are usually representation-based architecture, which can pre-compute representations offline and are therefore online efficient. Interaction-based models, although can achieve better performance, are mostly time-consuming and hard to be deployed online. Recently BERT has achieved significant progress on many NLP tasks including text matching, and it is of great value but also big challenge to deploy BERT to the e-commerce relevance task. To realize this goal, we propose ReprBERT, which has the advantages of both excellent performance and low latency, by distilling the interaction-based BERT model to a representation-based architecture. To reduce the performance decline, we investigate the key reasons and propose two novel interaction strategies to resolve the absence of representation interaction and low-level semantic interaction. Finally, ReprBERT can achieve only about 1.5% AUC loss from the interaction-based BERT, but has more than 10% AUC improvement compared to previous state-of-the-art representation-based models. ReprBERT has already been deployed on the search engine of Taobao and serving the entire search traffic, achieving significant gain of user experience and business profit.

The search engine plays a fundamental role in online e-commerce systems, to help users find the products they want from the massive product collections. Relevance is an essential requirement for e-commerce search, since showing products that do not match search query intent will degrade user experience. With the existence of vocabulary gap between user language of queries and seller language of products, measuring semantic relevance is necessary and neural networks are engaged to address this task. However, semantic relevance is different from click-through rate prediction in that no direct training signal is available. Most previous attempts learn relevance models from user click-through data that are cheap and abundant. Unfortunately, click behavior is noisy and misleading, which is affected by not only relevance but also factors including price, image and attractive titles. Therefore, it is challenging but valuable to learn relevance models from click-through data. In this paper, we propose a new relevance learning framework that concentrates on how to train a relevance model from the weak supervision of click-through data. Different from previous efforts that treat samples as either relevant or irrelevant, we construct more fine-grained samples for training. We propose a novel way to consider samples of different relevance confidence, and come up with a new training objective to learn a robust relevance model with desirable score distribution. The proposed model is evaluated on offline annotated data and online A/B testing, and it achieves both promising performance and high computational efficiency. The model has already been deployed online, serving the search traffic of Taobao for over a year.

For better user satisfaction and business effectiveness, Click-Through Rate (CTR) prediction is one of the most important tasks in E-commerce. It is often the case that users' interests different from their past routines may emerge or impressions such as promotional items may burst in a very short period. In essence, such changes relate to item evolution problem, which has not been investigated by previous studies. The state-of-the-art methods in the sequential recommendation, which use simple user behaviors, are incapable of modeling these changes sufficiently. It is because, in the user behaviors, outdated interests may exist and the popularity of an item over time is not well represented. To address these limitations, we introduce time-aware item behaviors for addressing the recommendation of emerging preference. The time-aware item behavior for an item is a set of users who interact with this item with timestamps. The rich interaction information of users for an item may help to model its evolution. In this work, we propose a CTR prediction model TIEN based on the time-aware item behavior. In TIEN, by leveraging the interaction time intervals, information of similar users in a short time interval helps identify the emerging user interest of the target user. By using the sequential time intervals, the item's popularity over time can be captured in evolutionary item dynamics. Noisy users who interact with items accidentally are further eliminated thus learning robust personalized item dynamics. To the best of our knowledge, this is the first study to the item evolution problem for E-commerce CTR prediction. We conduct extensive experiments on five real-world CTR prediction datasets. The results show that the TIEN model consistently achieves remarkable improvements to the state-of-the-art methods.

In this paper, we study the task of Query Auto-Completion (QAC), which is a very significant feature of modern search engines. In real industrial application, there always exist two major problems of QAC - weak personalization and unseen queries. To address these problems, we propose M2A, a multi-view multi-task attentive framework to learn personalized query auto-completion models. We propose a new Transformer-based hierarchical encoder to model different kinds of sequential behaviors, which can be seen as multiple distinct views of the user's searching history, and then a prefix-to-history attention mechanism is used to select the most relevant information to compose the final intention representation. To learn more informative representations, we propose to incorporate multi-task learning into the model training. Two different kinds of supervisory information provided by query logs are utilized at the same time by jointly training a CTR prediction model and a query generation model.

For better user experience and business effectiveness, Click-Through Rate (CTR) prediction has been one of the most important tasks in E-commerce. Although extensive CTR prediction models have been proposed, learning good representation of items from multimodal features is still less investigated, considering an item in E-commerce usually contains multiple heterogeneous modalities. Previous works either concatenate the multiple modality features, that is equivalent to giving a fixed importance weight to each modality; or learn dynamic weights of different modalities for different items through technique like attention mechanism. However, a problem is that there usually exists common redundant information across multiple modalities. The dynamic weights of different modalities computed by using the redundant information may not correctly reflect the different importance of each modality. To address this, we explore the complementarity and redundancy of modalities by considering modality-specific and modality-invariant features differently. We propose a novel Multimodal Adversarial Representation Network (MARN) for the CTR prediction task. A multimodal attention network first calculates the weights of multiple modalities for each item according to its modality-specific features. Then a multimodal adversarial network learns modality-invariant representations where a double-discriminators strategy is introduced. Finally, we achieve the multimodal item representations by combining both modality-specific and modality-invariant representations. We conduct extensive experiments on both public and industrial datasets, and the proposed method consistently achieves remarkable improvements to the state-of-the-art methods. Moreover, the approach has been deployed in an operational E-commerce system and online A/B testing further demonstrates the effectiveness.

Relevance is the core problem of a search engine, and one of the main challenges is the vocabulary gap between user queries and documents. This problem is more serious in e-commerce, because language in product titles is more professional. Query rewriting and semantic matching are two key techniques to bridge the semantic gap between them to improve relevance. Recently, deep neural networks have been successfully applied to the two tasks and enhanced the relevance performance. However, such approaches suffer from the sparseness of training data in e-commerce scenario. In this study, we investigate the instinctive connection between query rewriting and semantic matching tasks, and propose a co-training framework to address the data sparseness problem when training deep neural networks. We first build a huge unlabeled dataset from search logs, on which the two tasks can be considered as two different views of the relevance problem. Then we iteratively co-train them via labeled data generated from this unlabeled set to boost their performance simultaneously. We conduct a series of offline and online experiments on a real-world e-commerce search engine, and the results demonstrate that the proposed method improves relevance significantly.

Automatic pun generation is an interesting and challenging text generation task.Previous efforts rely on templates or laboriously manually annotated pun datasets, which heavily constrains the quality and diversity of generated puns.Since sequence-to-sequence models provide an effective technique for text generation, it is promising to investigate these models on the pun generation task.In this paper, we propose neural network models for homographic pun generation, and they can generate puns without requiring any pun data for training.We first train a conditional neural language model from a general text corpus, and then generate puns from the language model with an elaborately designed decoding algorithm.Automatic and human evaluations show that our models are able to generate homographic puns of good readability and quality.

Till now, neural abstractive summarization methods have achieved great success for single document summarization (SDS).However, due to the lack of large scale multi-document summaries, such methods can be hardly applied to multi-document summarization (MDS).In this paper, we investigate neural abstractive methods for MDS by adapting a state-of-the-art neural abstractive summarization model for SDS.We propose an approach to extend the neural abstractive model trained on large scale SDS data to the MDS task.Our approach only makes use of a small number of multi-document summaries for fine tuning.Experimental results on two benchmark DUC datasets demonstrate that our approach can outperform a variety of baseline neural models.

Quote is a language phenomenon of transcribing the statement of someone else, such as a proverb and a famous saying. An appropriate usage of quote usually equips the expression with more elegance and credibility. However, there are times when we are eager to stress our idea by citing a quote, while nothing relevant comes to mind. Therefore, it is exciting to have a recommender system which provides quote recommendations while we are writing. This article extends previous study of quote recommendation, the task that recommends the appropriate quote according to the context (i.e., the content occurring before and after the quote). In this article, a quote recommender system called QuoteRec is presented to tackle the task. We investigate two models to learn the vector representations of quotes and contexts, and then rank the candidate quotes based on the representations. The first model learns the quote representation according to the contexts of a quote. The second model is an extension of the neural network model in previous study, which learns the representation of a quote by concerning both its content and contexts. Experimental results demonstrate the effectiveness of the two models in learning the semantic representations of quotes, and the neural network model achieves state-of-the-art results on the quote recommendation task.

Abstractive summarization is the ultimate goal of document summarization research, but previously it is less investigated due to the immaturity of text generation techniques.Recently impressive progress has been made to abstractive sentence summarization using neural models.Unfortunately, attempts on abstractive document summarization are still in a primitive stage, and the evaluation results are worse than extractive methods on benchmark datasets.In this paper, we review the difficulties of neural abstractive document summarization, and propose a novel graph-based attention mechanism in the sequence-to-sequence framework.The intuition is to address the saliency factor of summarization, which has been overlooked by prior works.Experimental results demonstrate our model is able to achieve considerable improvement over previous neural abstractive models.The data-driven neural abstractive method is also competitive with state-of-the-art extractive methods.

Headline generation is a task of abstractive text summarization, and previously suffers from the immaturity of natural language generation techniques. Recent success of neural sentence summarization models shows the capacity of generating informative, fluent headlines conditioned on selected recapitulative sentences. In this paper, we investigate the extension of sentence summarization models to the document headline generation task. The challenge is that extending the sentence summarization model to consider more document information will mostly confuse the model and hurt the performance. In this paper, we propose a coarse-to-fine approach, which first identifies the important sentences of a document using document summarization techniques, and then exploits a multi-sentence summarization model with hierarchical attention to leverage the important sentences for headline generation. Experimental results on a large real dataset demonstrate the proposed approach significantly improves the performance of neural sentence summarization models on the headline generation task.

Quote is a language phenomenon of transcribing the saying of someone else. Proper usage of quote can usually make the statement more elegant and convincing. However, the ability of quote usage is usually limited by the amount of quotes one remembers or knows. Quote recommendation is a task of exploiting abundant quote repositories to help people make better use of quotes while writing. The task is different from conventional recommendation tasks due to the characteristic of quote. A pilot study has explored this task by using a learning to rank framework and manually designed features. However, it is still hard to model the meaning of a quote, which is an interesting and challenging problem. In this paper, we propose a neural network approach based on LSTMs to the quote recommendation task. We directly learn the distributed meaning representations for the contexts and the quotes, and then measure the relevance based on the meaning representations. In particular, we try to represent the words in quotes with specific embeddings, according to the contexts, topics and even author preferences of the quotes. Experimental results on a large dataset show that our proposed approach achieves the state-of-the-art performance and it outperforms several strong baselines.

In this paper, we propose and address a novel task of recommending quotes for writing. Quote is short for quotation, which is the repetition of someone else’s statement or thoughts. It is a common case in our writing when we would like to cite someone’s statement, like a proverb or a statement by some famous people, to make our composition more elegant or convincing. However, sometimes we are so eager to make a citation of quote somewhere, but have no idea about the relevant quote to express our idea. Because knowing or remembering so many quotes is not easy, it is exciting to have a system to recommend relevant quotes for us while writing. In this paper we tackle this appealing AI task, and build up a learning framework for quote recommendation. We collect abundant quotes from the Internet, and mine real contexts containing these quotes from large amount of electronic books, to build up a dataset for experiments. We explore the particular features of this task, and propose a few useful features to model the characteristics of quotes and the relevance of quotes to contexts. We apply a supervised learning to rank model to integrate multiple features. Experiment results show that, our proposed approach is appropriate for this task and it outperforms other recommendation methods.

The Weibo-oriented Chinese news summarization task aims to automatically generate a short summary for a given Chinese news article, and the short summary is used for news release and propagation on Sina Weibo. The length of the short summary is less than 140 Chinese characters. The task can be considered a special case of single document summarization. In this paper, we will introduce the evaluation dataset, the participating teams and the evaluation results. The dataset has been released publicly.

Manifold-ranking has proved to be an effective method for topic-focused multi-document summarization. As basic manifold-ranking based summarization method constructs the relationships between sentences simply by the bag-of-words cosine similarity, we believe a better similarity metric will further improve the effectiveness of manifold-ranking. In this paper, we propose a joint optimization framework, which integrates the manifold-ranking process with a similarity metric learning process. The joint framework aims at learning better sentence similarity scores and better sentence ranking scores simultaneously. Experiments on DUC datasets show the proposed joint method achieves better performance than the manifold-ranking baselines and several popular methods.