Product search systems must not only return relevant items but also understand users' implicit preferences beyond their explicit queries. For instance, when searching for "steak", most users implicitly prefer beef steak over equally relevant alternatives like pork steak. Predicting such engagement preferences presents a more complex challenge than traditional relevance modeling, as it requires capturing nuanced query-item relationships that reflect both relevance and user intent. To capture these nuances, we extend semantic understanding to engagement prediction by learning directly from query-item text with engagement labels as supervision rather than relying on historical engagement statistics as input. Our approach effectively captures users' implicit preferences across diverse query types, from tail queries where historical signals are sparse to broad queries where understanding latent intent is critical. Extensive experiments on Walmart's production search data demonstrate significant improvements over production model with a strong relevance foundation: +1.71% add-to-cart lift in interleaving tests and +0.33% in overall search sessions with add-to-cart. Our model is deployed in the production environment of Walmart.com.

Re-ranking plays a crucial role in product search by reassessing products from the primary retrieval system based on specific engagement and relevance criteria. While transformer-based models like the cross encoder have advanced the relevance of ranking models in recent years, a significant challenge arises from the high latency cost associated with running a cross encoder model at runtime. This challenge becomes more pronounced in the long-tail segment, where conventional techniques like caching prove ineffective. To tackle these issues, our paper introduces a scalable framework featuring a BERT-based cross encoder model for re-ranking, deployed in the Walmart search engine. We employ strategies such as intermediate representations, operator fusion, and vectorization to improve the inference latency of the cross encoder model. Furthermore, we provide a detailed discussion on the runtime implementation, highlighting key learnings and practical tricks that ensured minimal impact on response latency during production. Finally, we present the results of online experiments, including manual evaluation and interleaving test conducted on real-world e-commerce search traffic.

Supervisory signals are a critical resource for training learning to rank models. In many real-world search and retrieval scenarios, these signals may not be readily available or could be costly to obtain for some queries. The examples include domains where labeling requires professional expertise, applications with strong privacy constraints, and user engagement information that are too scarce. We refer to these scenarios as sparsely supervised queries which pose significant challenges to traditional learning to rank models. In this work, we address sparsely supervised queries by proposing a novel meta-learning to rank framework which leverages fast learning and adaption capability of meta-learning. The proposed approach accounts for the fact that different queries have different optimal parameters for their rankers, in contrast to traditional learning to rank models which only learn a global ranking model applied to all the queries. In consequence, the proposed method would yield significant advantages especially when new queries are of different characteristics with the training queries. Moreover, the proposed meta-learning to rank framework is generic and flexible. We conduct a set of comprehensive experiments on both public datasets and a real-world e-commerce dataset. The results demonstrate that the proposed meta-learning approach can significantly enhance the performance of learning to rank models with sparsely labeled queries.

In product search, the retrieval of candidate products before re-ranking is more mission critical and challenging than other search like web search, especially for tail queries, which have a complex and specific search intent. In this paper, we present a hybrid system for e-commerce search deployed at Walmart that combines traditional inverted index and embedding-based neural retrieval to better answer user tail queries. Our system significantly improved the relevance of the search engine, measured by both offline and online evaluations. The improvements were achieved through a combination of different approaches. We present a new technique to train the neural model at scale. and describe how the system was deployed in production with little impact on response time. We highlight multiple learnings and practical tricks that were used in the deployment of this system.

Product ranking is a crucial component for many e-commerce services. One of the major challenges in product search is the vocabulary mismatch between query and products, which may be a larger vocabulary gap problem compared to other information retrieval domains. While there is a growing collection of neural learning to match methods aimed specifically at overcoming this issue, they do not leverage the recent advances of large language models for product search. On the other hand, product ranking often deals with multiple types of engagement signals such as clicks, add-to-cart, and purchases, while most of the existing works are focused on optimizing one single metric such as click-through rate, which may suffer from data sparsity. In this work, we propose a novel end-to-end multi-task learning framework for product ranking with BERT to address the above challenges. The proposed model utilizes domain-specific BERT with fine-tuning to bridge the vocabulary gap and employs multi-task learning to optimize multiple objectives simultaneously, which yields a general end-to-end learning framework for product search. We conduct a set of comprehensive experiments on a real-world e-commerce dataset and demonstrate significant improvement of the proposed approach over the state-of-the-art baseline methods.

This paper presents a novel multiple anthropological Fisher kernel (MAFK) framework for kinship verification. The proposed MAFK framework, which goes beyond the Mahalanobis distance metric learning, integrates multiple anthropology inspired features and derives semantically meaningful similarities between images. The major novelty of this paper comes from the following three aspects. First, three new anthropology inspired features (AIF) are derived by extracting the AIF-SIFT, AIF-WLD and AIF-DAISY features on images that are enhanced by an anthropology inspired similarity enhancement method extended from the SIFT flow method. Second, a novel multiple anthropological Fisher kernel framework (MAFK) is proposed which combines multiple features and their metrics between images in a unified paradigm. The MAFK is optimized as a constrained, non-negative, and weighted variant of the sparse representation problem regularized by the criterion of pushing away the nearby non-kinship samples and pulling close the kinship samples. Third, a novel normalized kernel similarity measure (NKSM) is proposed by normalizing the MAFK with the fractional power transformation and L2 normalization. The feasibility of the proposed MAFK framework is assessed on two representative kinship data sets, namely the KinFaceW-I and the KinFaceW-II data sets. The experimental results show the effectiveness of the proposed method.

This paper presents an enhanced sparse coding method by exploiting both the generative and discriminative information in sparse representation model. Specifically, the proposed generative and discriminative sparse representation (GDSR) method integrates two new criteria, namely a discriminative criterion and a generative criterion, into the conventional sparse representation criterion. The generative criterion reveals the class conditional probability of each dictionary item by using the dictionary distribution coefficients which are derived by representing each dictionary item as a linear combination of the training samples. To further enhance the discriminative ability of the proposed method, a discriminative criterion is also applied using new localized within-class and between-class scatter matrices. Moreover, a novel GDSR based classification (GDSRc) method is proposed by utilizing both the derived sparse representation and the dictionary distribution coefficients. This hybrid method provides new insights, and leads to an effective representation and classification schema for improving the classification performance. The largest step size for learning the sparse representation is theoretically derived to address the convergence issues in the optimization procedure of the GDSR method. Extensive experimental results and analysis on several public classification datasets show the feasibility and effectiveness of the proposed method.

This paper presents an innovative sparse representation model using the complete marginal Fisher analysis (CMFA) framework for different challenging visual recognition tasks. First, a complete marginal Fisher analysis method is presented by extracting the discriminatory features in both the column space of the local samples based within the class scatter matrix and the null space of its transformed matrix. The rationale of extracting features in both spaces is to enhance the discriminatory power by further utilizing the null space, which is not accounted for in the marginal Fisher analysis method. Second, a discriminative sparse representation model is proposed by integrating a representation criterion such as the sparse representation and a discriminative criterion for improving the classification capability. In this model, the largest step size for learning the sparse representation is derived to address the convergence issues in optimization, and a dictionary screening rule is presented to purge the dictionary items with null coefficients for improving the computational efficiency. Experiments on some challenging visual recognition tasks using representative datasets, such as the Painting-91 dataset, the 15 scene categories dataset, the MIT-67 indoor scenes dataset, the Caltech 101 dataset, the Caltech 256 object categories dataset, the AR face dataset, and the extended Yale B dataset, show the feasibility of the proposed method.

Anthropology studies show that genetic features are inherited by children from their parents resulting in visual resemblance between them. This paper presents a novel SIFT flow based genetic Fisher vector feature (SF-GFVF) which enhances the facial genetic features for kinship verification. The proposed SF-GFVF feature is derived by applying a novel similarity enhancement method based on SIFT flow and learning an inheritable transformation on the Fisher vector feature so as to enhance and encode the genetic features of parent and child image in kinship relations. In particular, the similarity enhancement method is first presented by applying the SIFT flow algorithm to the densely sampled SIFT features in order to intensify the genetic features. Further analysis shows the relation of the extracted genetic features to anthropological results and discovers interesting patterns in different kinship relations. Finally, an inheritable transformation is applied to the enhanced Fisher vector feature which is learned with the criterion of minimizing the distance between kinship samples and maximizing the distance between non-kinship samples. Experimental results on the two representative kinship databases, namely the KinFace W-I and the Kinship W-II data sets show that the proposed method is able to outperform other popular methods.

Anthropology studies discover that some genetic related facial features, which are inherited by children from their parents, can be used for kinship verification. This paper investigates an important inheritable feature - color and presents a novel inheritable color space (InCS) and a generalized InCS (GInCS) framework with application to kinship verification. Specifically, a novel color similarity measure (CSM) is first defined. Second, based on this similarity measure, a new inheritable color space (InCS) is derived by balancing the criterion of minimizing the distance between kinship pairs and the criterion of maximizing the distance between non-kinship pairs. Unlike conventional color spaces, e.g. the RGB color space, the proposed InCS, which is learned automatically from the data, captures the inheritable information between parent and child. Third, theoretical and empirical analysis show that the proposed InCS exhibits the decorrelation property, which is positively related to the performance of kinship verification. Robustness to the illumination variation is also discussed. Fourth, a generalized InCS framework is presented to extend the InCS from the pixel level to the feature level for improving the performance and the robustness to illumination variation. The proposed InCS is evaluated on several popular datasets, namely the KinFaceW-I dataset, the KinFaceW-II dataset, the UB KinFace dataset, and the Cornell KinFace dataset. Experimental results show that the proposed InCS is able to (i) improve the conventional color spaces such as RGB, YUV, YIQ color spaces by a large margin, (ii) achieve robustness to the illumination variation, and (iii) outperforms other popular methods.

This paper presents a novel set of image features that encode the local, color, spatial, relative intensity information and gradient orientation of the painting image for painting artist classification, style classification as well as artist and style influence analysis. In particular, a new color DAISY Fisher vector (CD-FV) feature is first created by computing Fisher vectors on densely sampled DAISY features. Second, a color WLD-SIFT Fisher vector (CWS-FV) feature is developed by fusing Weber local descriptors (WLD) with Scale Invariant Feature Transform (SIFT) descriptors and Fisher vectors are computed on the fused WLD-SIFT features. Finally, an innovative color multi-fusion Fisher vector (CMFFV) feature is developed by integrating the Principal Component Analysis (PCA) features of CD-FV, CWS-FV and color SIFT-FV features. The effectiveness of the proposed CMFFV feature is assessed on the challenging Painting-91 dataset. Experimental results show that the proposed CMFFV feature is able to (i) achieve the state-of-the-art performance for painting artist classification, (ii) outperform other popular image descriptors, as well as (iii) discover the artist and style influence to understand their connections and evolution in different art movement periods.

This paper presents a novel general k nearest neighbour classifier (GKNNc) and a novel general nearest mean classifier (GNMc) for visual classification. Instead of treating the data equally, both GKNNc and GNMc assign a weight coefficient to each data. To achieve good performance, the conditions and properties of the weight coefficients for GKNNc and GNMc are further analysed. Then a sparse representation based method is proposed to derive the weight coefficients for both GKNNc and GNMc. Experimental results on several representative data sets, such as the Caltech 101 dataset and the MIT-67 indoor scenes dataset demonstrate the feasibility of the proposed methods.

This paper presents two novel discriminative dictionary learning models for sparse representation, namely the Fisher discriminative sparse model (FDSM) and the marginal Fisher discriminative sparse model (MFDSM). To learn the FDSM and the MFDSM efficiently and homogeneously, a general Fisher regularized model is further derived so that both of them can be learned without much modification. Experimental results on four popular databases, namely the extended Yale face database B, the AR face database, the 15 scenes dataset and the MIT-67 indoor scenes dataset show that the proposed method can improve upon other popular methods.