This paper develops the MUFIN technique for extreme classification (XC) tasks with millions of labels where data-points and labels are endowed with visual and textual de-scriptors. Applications of MUFIN to product-to-product recommendation and bid query prediction over several mil-lions of products are presented. Contemporary multi-modal methods frequently rely on purely embedding-based meth-ods. On the other hand, XC methods utilize classifier ar-chitectures to offer superior accuracies than embedding-only methods but mostly focus on text-based categorization tasks. MUFIN bridges this gap by reformulating multi-modal categorization as an XC problem with several mil-lions of labels. This presents the twin challenges of devel-oping multi-modal architectures that can offer embeddings sufficiently expressive to allow accurate categorization over millions of labels; and training and inference routines that scale logarithmically in the number of labels. MUFIN de-velops an architecture based on cross-modal attention and trains it in a modular fashion using pre-training and positive and negative mining. A novel product-to-product rec-ommendation dataset MM-AmazonTitles-300K containing over 300K products was curated from publicly available amazon.com listings with each product endowed with a title and multiple images. On the MM-AmazonTitles-300K and Polyvore datasets, and a dataset with over 4 million labels curated from click logs of the Bing search engine, MUFIN offered at least 3% higher accuracy than leading text-based, image-based and multi-modal techniques.

Machine learning techniques for developing industry-scale search engines have long been a prominent part of most domains and their online products. Search relevance algorithms are key components of products across different fields, including e-commerce, streaming services, and social networks. In this tutorial, we give an introduction to such large-scale search ranking systems, specifically focusing on deep learning techniques in this area. The topics we cover are the following: (1) Overview of search ranking systems in practice, including classical and machine learning techniques; (2) Introduction to sequential and language models in the context of search ranking; and (3) Knowledge distillation approaches for this area. For each of the aforementioned sessions, we first give an introductory talk and then go over an hands-on tutorial to really hone in on the concepts. We cover fundamental concepts using demos, case studies, and hands-on examples, including the latest Deep Learning methods that have achieved state-of-the-art results in generating the most relevant search results. Moreover, we show example implementations of these methods in python, leveraging a variety of open-source machine-learning/deep-learning libraries as well as real industrial data or open-source data.

In search and retrieval, a critical subtask is the classification of user search queries into predefined categories. Traditional supervised multi-class classification algorithms usually treat each category independently. In practical applications however, the categories have implicit relationships. Categories are organized as a tree-based taxonomy, which can be viewed as a graph. In this work, we explore a novel and systematic way of leveraging semantic information for improving taxonomy-based categorization. We propose a class of graph-based network structures, which we call Categorical Graph Neural Networks (CaGNN). CaGNNs leverage relationship information between neighbor categories and overlay the semantic information for each category, thus improving the performance of query categorization. The CaGNN framework can integrate a baseline categorizer with any Graph Neural Network, such as the commonly used Graph Attention Network and Graph Convolutional Network. Over a query categorization dataset of 2k categories and another ad title categorization dataset of 5k categories, CaGNN improves categorizers’ performance significantly compared to a baseline Deep Neural Network model without the CaGNN structure. Notably top 3 prediction recall increases from 90.15% to 91.40% for the ad title categorization task, for which is quite significant at over 90% level for more than 5k categories. By inspecting the learned category embeddings and the flow of message passing, we show that CaGNN effectively encapsulates useful graph structural information. Online A/B testing result shows that an ad ranking model with CaGNN-based features has increased ad click-through rate by 1.81% and reduced defect rate by 2.64%. The model has been deployed to production.

Automatic understanding of the product images, particularly the image attributes such as pattern, color, category, material, department, etc., has become a prerequisite for many downstream applications in the e-commerce ecosystem such as visually similar product retrieval and recommendation, publisher website image content monetization. With the recent advancement of deep learning techniques, multi-task deep neural networks (MTDNN) have been widely adopted for multi-class multi-label fashion image classification tasks. In this work, we propose a training technique Multi-Task Generative Adversarial Network (MT-GAN) to improve fashion image classification performance with an additional image translation task. We evaluated the proposed technique on two real-world fashion image datasets and experimental results show that both image classification and image translation benefit from this joint training setting. Specifically the image classifier, even though trained with a smaller amount of data and fewer types of labeled information, manages to outperform existing image classification models such as WTBI and DARN by 54% and 20% respectively for category classification on DeepFashion benchmark. Two main contributors to the superior performance of the image classifier trained as the discriminator in the proposed MT-GAN framework are: 1) the classifier leverages additional amount of data and the regularization effect by taking on the task of a traditional discriminator. 2) The classifier takes not only the RGB image but also the conditional input image shared between the generator and itself as its input, boosting the classification accuracy and recall by 18%. The proposed MT-GAN training technique is GAN-agnostic and can be applied to various state-of-the-art designs of generator and discriminator as well.

Deep Reinforcement Learning has been applied in a number of fields to directly optimize non-differentiable reward functions, including in sequence to sequence settings using Self Critical Sequence Training (SCST). Previously, SCST has primarily been applied to bring conditional language models closer to the distribution of their training set, as in traditional neural machine translation and abstractive summarization. We frame the generation of search engine text ads as a sequence to sequence problem, and consider two related goals: to generate ads similar to those a human would write, and to generate ads with high click-through rates. We jointly train a model to minimize cross-entropy on an existing corpus of Landing Page/Text Ad pairs using typical sequence to sequence training techniques while also optimizing the expected click-through rate (CTR) as predicted by an existing oracle model using SCST. Through joint training we achieve a 6.7% increase in expected CTR without a meaningful drop in ROUGE score. Human experiments demonstrate that SCST training produces significantly more attractive ads without reducing grammatical quality.

Information extraction and user intention identification is a central topic in modern query understanding and recommendation systems. In this paper, we propose DeepProbe, a generic information-directed interaction framework which is built around an attention-based sequence to sequence (seq2seq) recurrent neural network. DeepProbe can rephrase, evaluate, and even actively ask questions, leveraging the generative ability and likelihood estimation made possible by seq2seq models. DeepProbe makes decisions based on a derived uncertainty (entropy) measure conditioned on user inputs, possibly with multiple rounds of interactions. Three applications, namely a rewritter, a relevance scorer and a chatbot for ad recommendation, were built around DeepProbe, with the first two serving as precursory building blocks for the third. We first use the seq2seq model in DeepProbe to rewrite a user query into one of standard query form, which is submitted to an ordinary recommendation system. Secondly, we evaluate DeepProbe's seq2seq model-based relevance scoring. Finally, we build a chatbot prototype capable of making active user interactions, which can ask questions that maximize information gain, allowing for a more efficient user intention idenfication process. We evaluate first two applications by 1) comparing with baselines by BLEU and AUC, and 2) human judge evaluation. Both demonstrate significant improvements compared with current state-of-the-art systems, proving their values as useful tools on their own, and at the same time laying a good foundation for the ongoing chatbot application.

In this paper, we investigate the use of recurrent neural networks (RNNs) in the context of search-based online advertising. We use RNNs to map both queries and ads to real valued vectors, with which the relevance of a given (query, ad) pair can be easily computed. On top of the RNN, we propose a novel attention network, which learns to assign attention scores to different word locations according to their intent importance (hence the name DeepIntent). The vector output of a sequence is thus computed by a weighted sum of the hidden states of the RNN at each word according their attention scores. We perform end-to-end training of both the RNN and attention network under the guidance of user click logs, which are sampled from a commercial search engine. We show that in most cases the attention network improves the quality of learned vector representations, evaluated by AUC on a manually labeled dataset. Moreover, we highlight the effectiveness of the learned attention scores from two aspects: query rewriting and a modified BM25 metric. We show that using the learned attention scores, one is able to produce sub-queries that are of better qualities than those of the state-of-the-art methods. Also, by modifying the term frequency with the attention scores in a standard BM25 formula, one is able to improve its performance evaluated by AUC.

This paper investigates the usefulness of segmental phoneme-dynamics for classification of speaking styles. We modeled transition details based on the phoneme sequences emitted by a speech recognizer, using data obtained from a recording of 39 depressed patients with 7 different speaking styles- nor-mal, pressured, slurred, stuttered, flat, slow and fast speech. We designed and compared two set of phoneme models: a lan-guage model treating each phoneme as a word unit (one for each style) and a context-dependent phoneme duration model based on Gaussians for each speaking style considered. The experi-ments showed that language modeling at the phoneme level per-formed better than the duration model. We also found that better performance can be obtained by user normalization. To see the complementary effect of the phoneme-based models, the classi-fiers were combined at a decision level with a Hidden Markov Model (HMM) classifier built from spectral features. The im-provement was 5.7 % absolute (10.4 % relative), reaching 60.3% accuracy in 7-class and 71.0 % in 4-class classification. Index Terms: speaking styles, language model, phoneme, du-ration