Recommender systems have become integral to many online services, leveraging user data to provide personalized recommendations. However, as these systems grow in complexity, understanding the rationale behind their recommendations becomes increasingly difficult. Explainable Artificial Intelligence (XAI) has emerged as a crucial field addressing this challenge, particularly in ensuring transparency and trustworthiness in automated decision-making processes. In this article, we introduce Learning to eXplain Recommendations (LXR) , a scalable, model-agnostic framework designed to generate counterfactually correct explanations for recommender systems. LXR generates explanations for recommendations produced by any differentiable recommender system. By leveraging both factual and counterfactual loss terms, LXR offers robust, accurate, and computationally efficient explanations that reflect the model’s internal decision-making process. A key feature of LXR is its focus on the factual correctness of explanations through counterfactual reasoning, bridging the gap between plausible and accurate explanations. Unlike traditional approaches that rely on exhaustive perturbations of user data, LXR uses a self-supervised learning method to generate explanations efficiently, without sacrificing accuracy. LXR operates in two stages: a pre-training step and a novel Inference-Time Fine-tuning (ITF) step that refines explanations at the individual recommendation level, significantly improving accuracy with minimal computational overhead. Additionally, LXR is applied to hybrid recommender models incorporating demographic data, demonstrating its versatility across real-world scenarios. Finally, we also showcase LXR’s ability to explain recommendations at various ranks within a user’s recommendation list. As a secondary contribution, we introduce several novel evaluation metrics, inspired by saliency maps from computer vision, to rigorously assess the counterfactual correctness of explanations in recommender systems. Our results demonstrate that LXR sets a new benchmark for explainability, providing accurate, transparent, and interpretable explanations. The code is available on our GitHub repository: https://github.com/DeltaLabTLV/LXR_ .

Counterfactual evaluation provides a promising framework for assessing explanation fidelity in recommender systems, but perturbation metrics adapted from computer vision suffer three key limitations: (1) they conflate explaining and contradictory features, (2) they average over entire user histories instead of prioritizing concise, high-impact explanations, and (3) they use fixed-percentage perturbations, leading to inconsistencies across users.We introduce refined counterfactual metrics that focus on the most relevant explaining features, exclude contradictory elements, and assess fidelity at a fixed explanation length, ensuring a more consistent and interpretable evaluation.Our code is at: https:// github.com/DeltaLabTLV/FidelityMetrics4XRec

Knowledge Graph (KG)-based recommender systems utilize both Collaborative Filtering (CF) data and informative KG data to improve prediction accuracy. Path-based KG methods are a family of KG-based recommendation models that explore the interlinks within a knowledge graph in order to enhance the connectivity between users and items with rich complementary information. A key advantage of path-based KG methods stems from their ability to enable intuitive explanations naturally. In this work, we present a novel path-based algorithm that employs neural attention in order to better extract the relevant information from the unified graph. Evaluations based on public KG recommendation datasets indicate a clear advantage to the proposed method compared to state-of-the-art path-based alternatives. Furthermore, we show that this advantage also extends to cold items where a better utilization of the KG leads to improved predictions in cases where no CF data is available. Finally, by performing attention-score analysis, we demonstrate the ability of our approach to provide better interpretability into the model’s inner workings as well as extract more intuitive explanations. The code for this work is publicly available on GitHub: https://github.com/DeltaLabTLV/KPAR .

This empirical study addresses the problem of Next Basket Repurchase Recommendation (NBRR), an often overlooked aspect of Next Basket Recommendation (NBR). While NBR aims to suggest items for a user’s next basket based on their prior basket history, NBRR focuses solely on recommending items previously purchased by the user. Despite the common ground between NBR and NBRR, the latter requires a distinct approach. In this paper, we survey recent developments in the fields of NBR and NBRR, emphasizing the different strategies employed for these closely related challenges. In addition, we review the common characteristics of users’ repurchase patterns, which characterize the NBRR problem. Building on these insights, we introduce a novel hyper-convolutional model tailored to capture behavioral patterns associated with repeated purchases. To evaluate its effectiveness, we conduct experiments on three publicly available datasets, offering a comprehensive analysis across three levels of granularity: user-level, order-level, and item-level. Our analysis illuminates the conditions under which the model excels and identifies scenarios where it may encounter challenges. This research contributes valuable insights into enhancing repurchase recommendation systems and advancing the understanding of user purchase behavior in general.

We present Learning Attributions (LA), a novel method for explaining language models. The core idea behind LA is to train a dedicated attribution model that functions as a surrogate explainer for the language model. This attribution model is designed to identify which tokens are most influential in driving the model's predictions. By optimizing the attribution model to mask the minimal amount of information necessary to induce substantial changes in the language model's output, LA provides a mechanism to understand which tokens in the input are critical for the model's decisions. We demonstrate the effectiveness of LA across several language models, highlighting its superiority over multiple state-of-the-art explanation methods across various datasets and evaluation metrics.

In the field of recommender systems, explainability remains a pivotal yet challenging aspect. To address this, we introduce the Learning to eXplain Recommendations (LXR) framework, a post-hoc, model-agnostic approach designed for providing counterfactual explanations. LXR is compatible with any differentiable recommender algorithm and scores the relevance of user data in relation to recommended items. A distinctive feature of LXR is its use of novel self-supervised counterfactual loss terms, which effectively highlight the most influential user data responsible for a specific recommended item. Additionally, we propose several innovative counterfactual evaluation metrics specifically tailored for assessing the quality of explanations in recommender systems. Our code is available on our GitHub repository: https://github.com/DeltaLabTLV/LXR.

This paper presents Deep Integrated Explanations (DIX) - a universal method for explaining vision models. DIX generates explanation maps by integrating information from the intermediate representations of the model, coupled with their corresponding gradients. Through an extensive array of both objective and subjective evaluations spanning diverse tasks, datasets, and model configurations, we showcase the efficacy of DIX in generating faithful and accurate explanation maps, while surpassing current state-of-the-art methods. Our code is available at: https://github.com/dix-cikm23/dix

Transcriptions of phone calls hold significant value in sales, customer service, healthcare, law enforcement, and more. However, analyzing recorded conversations can be a time-consuming process, especially for complex dialogues. In Microsoft Dynamics 365 Sales, a novel system, named GPT-Calls, is applied for efficient and accurate topic-based call segmentation. GPT-Calls comprises offline and online phases. In the offline phase, the system leverages a GPT model to generate synthetic sentences and extract anchor vectors for predefined topics. This phase, performed once on a given topic list, significantly reduces the computational burden. The online phase scores the similarity between the transcribed conversation and the topic anchors from the offline phase, followed by time domain analysis to group utterances into segments and tag them with topics. The GPT-Calls scheme offers an accurate and efficient approach to call segmentation and topic extraction, eliminating the need for labeled data. It is a versatile solution applicable to various industry domains. GPT-Calls operates in production under Dynamics 365 Sales Conversation Intelligence, applied to real sales conversations from diverse Dynamics 365 Sales tenants, streamlining call analysis, and saving time and resources while ensuring accuracy and effectiveness.

We present Learning to Explain (LTX), a model-agnostic framework designed for providing post-hoc explanations for vision models. The LTX framework introduces an “explainer” model that generates explanation maps, highlighting the crucial regions that justify the predictions made by the model being explained. To train the explainer, we employ a two-stage process consisting of initial pretraining followed by per-instance finetuning. During both stages of training, we utilize a unique configuration where we compare the explained model’s prediction for a masked input with its original prediction for the unmasked input. This approach enables the use of a novel counterfactual objective, which aims to anticipate the model’s output using masked versions of the input image. Importantly, the LTX framework is not restricted to a specific model architecture and can provide explanations for both Transformer-based and convolutional models. Through our evaluations, we demonstrate that LTX significantly outperforms the current state-of-the-art in explainability across various metrics. Our code is available at: https://github.comLTX-CodeLTX

We introduce Stochastic Integrated Explanations (SIX) - a general method for explaining predictions made by vision models. SIX employs stochastic integration on the internal representations across different network layers, producing explanation maps at various scales. The primary innovation of SIX is the introduction of randomness to the integration process by modeling the baseline representation as a random tensor. Through iterative sampling from the baseline distribution, SIX generates a diverse set of explanation maps, allowing the selection of the best-performing map based on a specific metric of interest. Extensive evaluations on various model architectures showcase the superior performance of SIX compared to state-of-the-art explanation methods, affirming its effectiveness across multiple metrics. Our code is available at: https://github.com/six-icdm/six

The problem of Next Basket Recommendation (NBR) addresses the challenge of recommending items for the next basket of a user, based on her sequence of prior baskets. In this paper, we focus on a variation of this problem in which we aim to predict repurchases, i.e. we wish to recommend a user only items she had purchased before. We coin this problem Next Basket Repurchase Recommendation (NBRR). Over the years, a variety of models have been proposed to address the problem of NBR, however, the problem of NBRR has been overlooked. Although being highly related problems, which are often solved by the same methods, the problem of repurchase recommendation calls for a different approach. In this paper, we share insights from our experience of facing the challenge of NBRR. In light of these insights, we propose a novel hyper-convolutional model to leverage the behavioral patterns of repeated purchases. We demonstrate the effectiveness of the proposed model on three publicly available datasets, where it is shown to outperform other existing methods across multiple metrics.

Recently, there has been growing interest in the ability of Transformer-based models to produce meaningful embeddings of text with several applications, such as text similarity. Despite significant progress in the field, the explanations for similarity predictions remain challenging, especially in unsupervised settings. In this work, we present an unsupervised technique for explaining paragraph similarities inferred by pre-trained BERT models. By looking at a pair of paragraphs, our technique identifies important words that dictate each paragraph’s semantics, matches between the words in both paragraphs, and retrieves the most important pairs that explain the similarity between the two. The method, which has been assessed by extensive human evaluations and demonstrated on datasets comprising long and complex paragraphs, has shown great promise, providing accurate interpretations that correlate better with human perceptions.

We present Gradient Activation Maps (GAM) - a machinery for explaining predictions made by visual similarity and classification models. By gleaning localized gradient and activation information from multiple network layers, GAM offers improved visual explanations, when compared to existing alternatives. The algorithmic advantages of GAM are explained in detail, and validated empirically, where it is shown that GAM outperforms its alternatives across various tasks and datasets.

We present Variational Bayesian Network (VBN) - a novel Bayesian entity representation learning model that utilizes hierarchical and relational side information and is particularly useful for modeling entities in the "long-tail'', where the data is scarce. VBN provides better modeling for long-tail entities via two complementary mechanisms: First, VBN employs informative hierarchical priors that enable information propagation between entities sharing common ancestors. Additionally, VBN models explicit relations between entities that enforce complementary structure and consistency, guiding the learned representations towards a more meaningful arrangement in space. Second, VBN represents entities by densities (rather than vectors), hence modeling uncertainty that plays a complementary role in coping with data scarcity. Finally, we propose a scalable Variational Bayes optimization algorithm that enables fast approximate Bayesian inference. We evaluate the effectiveness of VBN on linguistic, recommendations, and medical inference tasks. Our findings show that VBN outperforms other existing methods across multiple datasets, and especially in the long-tail.

Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.

Modern-day recommender systems are often based on learning representations in a latent vector space that encode user and item preferences. In these models, each user/item is represented by a single vector and user-item interactions are modeled by some function over the corresponding vectors. This paradigm is common to a large body of collaborative filtering models that repeatedly demonstrated superior results. In this work, we break away from this paradigm and present ACF: Anchor-based Collaborative Filtering. Instead of learning unique vectors for each user and each item, ACF learns a spanning set of anchor-vectors that commonly serve both users and items. In ACF, each anchor corresponds to a unique "taste'' and users/items are represented as a convex combination over the spanning set of anchors. Additionally, ACF employs two novel constraints: (1) exclusiveness constraint on item-to-anchor relations that encourages each item to pick a single representative anchor, and (2) an inclusiveness constraint on anchors-to-items relations that encourages full utilization of all the anchors. We compare ACF with other state-of-the-art alternatives and demonstrate its effectiveness on multiple datasets.

A major challenge in collaborative filtering methods is how to produce recommendations for cold items (items with no ratings), or integrate cold items into an existing catalog. Over the years, a variety of hybrid recommendation models have been proposed to address this problem by utilizing items’ metadata and content along with their ratings or usage patterns. In this work, we wish to revisit the cold start problem in order to draw attention to an overlooked challenge: the ability to integrate and balance between (regular) warm items and completely cold items. In this case, two different challenges arise: (1) preserving high-quality performance on warm items, while (2) learning to promote cold items to relevant users. First, we show that these two objectives are in fact conflicting, and the balance between them depends on the business needs and the application at hand. Next, we propose a novel hybrid recommendation algorithm that bridges these two conflicting objectives and enables a harmonized balance between preserving high accuracy for warm items while effectively promoting completely cold items. We demonstrate the effectiveness of the proposed algorithm on movies, apps, and articles recommendations, and provide an empirical analysis of the cold-warm trade-off.

Learning item representations is a key building block in recommender systems research. However, representations often suffer from the cold start problem - a well-known problem in which rare items in the tail of the distribution face insufficient data yielding inadequate representations. In this work, we present a novel hybrid recommender that supports the utilization of hierarchical content-based information to mitigate the cold start problem. In particular, we assume a taxonomy of item tags in which every item is associated with several `parent' tags and the tags themselves can be associated with several `parent' tags in a hierarchical manner. Our model learns item representations that are guided by the `parent' tags of each item which allows propagating relevant information between items sharing the same hierarchy. In addition, the tags are modeled using tag representations that allow propagating information between any two tags that share a common ancestor. Due to space limitation, we focus this work on a recommendations task, however the same approach can be utilized for general representation learning e.g. language models.

Two main challenges in recommender systems are modeling users with heterogeneous taste, and providing explainable recommendations. In this paper, we propose the neural Attentive Multi-Persona Collaborative Filtering (AMP-CF) model as a unified solution for both problems. AMP-CF breaks down the user to several latent ‘personas’ (profiles) that identify and discern the different tastes and inclinations of the user. Then, the revealed personas are used to generate and explain the final recommendation list for the user. AMP-CF models users as an attentive mixture of personas, enabling a dynamic user representation that changes based on the item under consideration. We demonstrate AMP-CF on five collaborative filtering datasets from the domains of movies, music, video games and social networks. As an additional contribution, we propose a novel evaluation scheme for comparing the different items in a recommendation list based on the distance from the underlying distribution of “tastes” in the user’s historical items. Experimental results show that AMP-CF is competitive with other state-of-the-art models. Finally, we provide qualitative results to showcase the ability of AMP-CF to explain its recommendations.

In Recommender Systems research, algorithms are often characterized as either Collaborative Filtering (CF) or Content Based (CB). CF algorithms are trained using a dataset of user preferences while CB algorithms are typically based on item profiles. These approaches harness different data sources and therefore the resulting recommended items are generally very different. This paper presents the CB2CF, a deep neural multiview model that serves as a bridge from items content into their CF representations. CB2CF is a "real-world" algorithm designed for Microsoft Store services that handle around a billion users worldwide. CB2CF is demonstrated on movies and apps recommendations, where it is shown to outperform an alternative CB model on completely cold items.

Maximizing sales and revenue is an important goal of online commercial retailers. Recommender systems are designed to maximize users' click or purchase probability, but often disregard users' eventual satisfaction with purchased items. As result, such systems promote items with high appeal at the selling stage (e.g. an eyecatching presentation) over items that would yield more satisfaction to users in the long run. This work presents a novel unified model that considers both goals and can be tuned to balance between them according to the needs of the business scenario.

Microsoft Windows is the most popular operating system (OS) for personal computers (PCs). With hundreds of millions of users, its app marketplace, Windows Store, is one of the largest in the world. As such, special considerations are required in order to improve online computational efficiency and response times.

A decade has passed since the seminal Netflix Prize competition and Collaborative Filtering (CF) models are still at the forefront of Recommender System research. Significant progress has been achieved over this time, yet key aspects of the basic problem formulation have not been seriously challenged. Most state-of-the-art models still assume a supervised model in which the ultimate goal is to predict future user-item interactions based on the generalization of historical data.

This paper describes an algorithm designed for Microsoft's Groove music service, which serves millions of users world wide. We consider the problem of automatically generating personalized music playlists based on queries containing a ``seed'' artist and the listener's user ID. Playlist generation may be informed by a number of information sources including: user specific listening patterns, domain knowledge encoded in a taxonomy, acoustic features of audio tracks, and overall popularity of tracks and artists. The importance assigned to each of these information sources may vary depending on the specific combination of user and seed~artist.

Determinantal point processes (DPPs) are an emerging model for encoding probabilities over subsets, such as shopping baskets, selected from a ground set, such as an item catalog. They have recently proved to be appealing models for a number of machine learning tasks, including product recommendation. DPPs are parametrized by a positive semi-definite kernel matrix. Prior work has shown that using a low-rank factorization of this kernel provides scalability improvements that open the door to training on large-scale datasets and computing online recommendations, both of which are infeasible with standard DPP models that use a full-rank kernel. A low-rank DPP model can be trained using an optimization-based method, such as stochastic gradient ascent, to find a point estimate of the kernel parameters, which can be performed efficiently on large-scale datasets. However, this approach requires careful tuning of regularization parameters to prevent overfitting and provide good predictive performance, which can be computationally expensive. In this paper we present a Bayesian method for learning a low-rank factorization of this kernel, which provides automatic control of regularization. We show that our Bayesian low-rank DPP model can be trained efficiently using stochastic gradient Hamiltonian Monte Carlo (SGHMC). Our Bayesian model generally provides better predictive performance on several real-world product recommendation datasets than optimization-based low-rank DPP models trained using stochastic gradient ascent, and better performance than several state-of-the art recommendation methods in many cases.

Most Collaborative Filtering (CF) algorithms are optimized using a dataset of isolated user-item tuples. However, in commercial applications recommended items are usually served as an ordered list of several items and not as isolated items. In this setting, inter-item interactions have an effect on the list's Click-Through Rate (CTR) that is unaccounted for using traditional CF approaches. Most CF approaches also ignore additional important factors like click propensity variation, item fatigue, etc. In this work, we introduce the list recommendation problem. We present useful insights gleaned from user behavior and consumption patterns from a large scale real world recommender system. We then propose a novel two-layered framework that builds upon existing CF algorithms to optimize a list's click probability. Our approach accounts for inter-item interactions as well as additional information such as item fatigue, trendiness patterns, contextual information etc. Finally, we evaluate our approach using a novel adaptation of Inverse Propensity Scoring (IPS) which facilitates off-policy estimation of our method's CTR and showcases its effectiveness in real-world settings.

A prominent approach in collaborative filtering based recommender systems is using dimensionality reduction (matrix factorization) techniques to map users and items into low-dimensional vectors. In such systems, a higher inner product between a user vector and an item vector indicates that the item better suits the user's preference. Traditionally, retrieving the most suitable items is done by scoring and sorting all items. Real world online recommender systems must adhere to strict response-time constraints, so when the number of items is large, scoring all items is intractable.

Most recommenders research aims at personalized systems, which suggest items based on user profiles. However, in reality many systems deal with item-oriented recommendations. In such setups, given a single item of interest, the system needs to provide other related items, following patterns like "people who liked this also liked...".

We present a matrix factorization model inspired by challenges we encountered while working on the Xbox movies recommendation system. The item catalog in a recommender system is typically equipped with meta-data features in the form of labels. However, only part of these features are informative or useful with regard to collaborative filtering. By incorporating a novel sparsity prior on feature parameters, the model automatically discerns and utilizes informative features while simultaneously pruning non-informative features.

We study the problem of scoring and selecting content-based features for a collaborative filtering (CF) recommender system. Content-based features play a central role in mitigating the ``cold start'' problem in commercial recommenders. They are also useful in other related tasks, such as recommendation explanation and visualization. However, traditional feature selection methods do not generalize well to recommender systems. As a result, commercial systems typically use manually crafted and selected features. This work presents a framework for automated selection of informative content-based features, that is independent of the type of recommender system or the type of features. We evaluate on recommenders from different domains: books, movies and smart-phone apps, and show effective results on each. In addition, we show how to use the proposed methods to generate meaningful features from text.

Project Sage is Microsoft's all-purpose recommender system, designed and deployed as an ultra-high scale cloud service. Sage focuses on both state of the art research and high scale robust implementation. In the research front, we demonstrate new pre-processing and cleaning techniques, a novel probabilistic matrix factorization model for implicit one-class data, and a relatively new evaluation framework. In the engineering front, we present a working service deployed on the Microsoft Azure cloud, which provides easy-to-use interfaces to integrate a recommendation service into any website.

The bane of one-class collaborative filtering is interpreting and modelling the latent signal from the missing class. In this paper we present a novel Bayesian generative model for implicit collaborative filtering. It forms a core component of the Xbox Live architecture, and unlike previous approaches, delineates the odds of a user disliking an item from simply being unaware of it. The latent signal is treated as an unobserved random graph connecting users with items they might have encountered. We demonstrate how large-scale distributed learning can be achieved through a combination of stochastic gradient descent and mean field variational inference over random graph samples. A fine-grained comparison is done against a state of the art baseline on real world data.

Low-rank Matrix Factorization (MF) methods provide one of the simplest and most effective approaches to collaborative filtering. This paper is the first to investigate the problem of efficient retrieval of recommendations in a MF framework. We reduce the retrieval in a MF model to an apparently simple task of finding the maximum dot-product for the user vector over the set of item vectors. However, to the best of our knowledge the problem of efficiently finding the maximum dot-product in the general case has never been studied. To this end, we propose two techniques for efficient search -- (i) We index the item vectors in a binary spatial-partitioning metric tree and use a simple branch and-bound algorithm with a novel bounding scheme to efficiently obtain exact solutions. (ii) We use spherical clustering to index the users on the basis of their preferences and pre-compute recommendations only for the representative user of each cluster to obtain extremely efficient approximate solutions. We obtain a theoretical error bound which determines the quality of any approximate result and use it to control the approximation. Both these simple techniques are fairly independent of each other and hence are easily combined to further improve recommendation retrieval efficiency. We evaluate our algorithms on real-world collaborative-filtering datasets, demonstrating more than ×7 speedup (with respect to the naive linear search) for the exact solution and over ×250 speedup for approximate solutions by combining both techniques.

A recent addition to Microsoft's Xbox Live Marketplace is a recommender system which allows users to explore both movies and games in a personalized context. The system largely relies on implicit feedback, and runs on a large scale, serving tens of millions of daily users. We describe the system design, and review the core recommendation algorithm.

In the past decade large scale recommendation datasets were published and extensively studied. In this work we describe a detailed analysis of a sparse, large scale dataset, specifically designed to push the envelope of recommender system models. The Yahoo! Music dataset consists of more than a million users, 600 thousand musical items and more than 250 million ratings, collected over a decade. It is characterized by three unique features: First, rated items are multi-typed, including tracks, albums, artists and genres; Second, items are arranged within a four level taxonomy, proving itself effective in coping with a severe sparsity problem that originates from the unusually large number of items (compared to, e.g., movie ratings datasets). Finally, fine resolution timestamps associated with the ratings enable a comprehensive temporal and session analysis. We further present a matrix factorization model exploiting the special characteristics of this dataset. In particular, the model incorporates a rich bias model with terms that capture information from the taxonomy of items and different temporal dynamics of music ratings. To gain additional insights of its properties, we organized the KddCup-2011 competition about this dataset. As the competition drew thousands of participants, we expect the dataset to attract considerable research activity in the future.

Record label companies would like to identify potential artists as early as possible in their careers, before other companies approach the artists with competing contracts. The vast number of candidates makes the process of identifying the ones with high success potential time consuming and laborious. This paper demonstrates how datamining of P2P query strings can be used in order to mechanize most of this detection process. Using a unique intercepting system over the Gnutella network, we were able to capture an unprecedented amount of geographically identified (geo-aware) queries, allowing us to investigate the diffusion of music related queries in time and space. Our solution is based on the observation that emerging artists, especially rappers, have a discernible stronghold of fans in their hometown area, where they are able to perform and market their music. In a file sharing network, this is reflected as a delta function spatial distribution of content queries. Using this observation, we devised a detection algorithm for emerging artists, that looks for performers with sharp increase in popularity in a small geographic region though still unnoticable nation wide. The algorithm can suggest a short list of artists with breakthrough potential, from which we showed that about 30% translate the potential to national success.