The rapid advancement of multimodal large language models has enabled agents to operate mobile devices by directly interacting with graphical user interfaces, opening new possibilities for mobile automation. However, real-world mobile tasks are often complex and allow for multiple valid solutions. This contradicts current mobile agent evaluation standards: offline static benchmarks can only validate a single predefined ''golden path'', while online dynamic testing is constrained by the complexity and non-reproducibility of real devices, making both approaches inadequate for comprehensively assessing agent capabilities. To bridge the gap between offline and online evaluation and enhance testing stability, this paper introduces a novel graph-structured benchmarking framework. By modeling the finite states observed during real-device interactions, it achieves static simulation of dynamic behaviors. Building on this, we develop ColorBench, a benchmark focused on complex long-horizon tasks. It supports evaluation of multiple valid solutions, subtask completion rate statistics, and atomic-level capability analysis. ColorBench contains 175 tasks (74 single-app, 101 cross-app) with an average length of over 13 steps. Each task includes at least two correct paths and several typical error paths, enabling quasi-dynamic interaction.

Recommender system plays a pervasive role in today’s online services, yet its closed-loop nature, i.e., training and deploying within a specific closed domain, constrains its access to open-world knowledge. Recently, the emergence of large language models (LLMs) has shown promise in bridging this gap by encoding extensive world knowledge and demonstrating advanced reasoning capabilities. However, previous attempts to directly implement LLMs as recommenders fall short in meeting the demanding requirements of industrial recommender systems, particularly in terms of online inference latency and offline resource efficiency. In this work, we propose an Open-World R ecommendation Framework with E fficient and Deployable K nowledge I nfusion from Large Language Models, dubbed REKI , to acquire two types of external knowledge about users and items from LLMs. Specifically, we introduce factorization prompting to elicit accurate knowledge reasoning on user preferences and items. With factorization prompting, we develop individual knowledge extraction and collective knowledge extraction tailored for different scales of recommendation scenarios, effectively reducing offline resource consumption. Subsequently, the generated user and item knowledge undergoes efficient transformation and condensation into augmented vectors through a hybridized expert-integrated network , ensuring its compatibility with the recommendation task. The obtained vectors can then be directly used to enhance the performance of any conventional recommendation model. We also ensure efficient inference by preprocessing and prestoring the knowledge from the LLM. Extensive experiments demonstrate that REKI significantly outperforms the state-of-the-art baselines and is compatible with a diverse array of recommendation algorithms and tasks. Now, REKI has been deployed to Huawei’s news and music recommendation platforms and gained a 7% and 1.99% improvement during the online A/B test.

Large Language Models (LLMs) have exhibited significant promise in recommender systems by empowering user profiles with their extensive world knowledge and superior reasoning capabilities. However, LLMs face challenges like unstable instruction compliance, modality gaps, and high inference latency, leading to textual noise and limiting their effectiveness in recommender systems. To address these challenges, we propose UserIP-Tuning, which uses prompt-tuning to infer user profiles. It integrates the causal relationship between user profiles and behavior sequences into LLMs' prompts. It employs Expectation Maximization (EM) to infer the embedded latent profile, minimizing textual noise by fixing the prompt template. Furthermore, a profile quantization codebook bridges the modality gap by categorizing profile embeddings into collaborative IDs pre-stored for online deployment. This improves time efficiency and reduces memory usage. Experiments show that UserIP-Tuning outperforms state-of-the-art recommendation algorithms. An industry application confirms its effectiveness, robustness, and transferability. The presented solution has been deployed in Huawei AppGallery's Explore page since May 2025, serving 2 million daily active users, delivering significant improvements in real-world recommendation scenarios. The code is publicly available for replication at https://github.com/Applied-Machine-Learning-Lab/UserIP-Tuning.

Interactive recommender systems (IRSs) aim to meet user needs through natural language dialogues, optimizing recommendations with minimal interactions. Typically, IRSs are based on large language models (LLMs). Existing methods generally consist of two stages: decision-making (deciding whether to recommend or ask clarification questions) and action execution (generating recommendations or clarification questions). These methods usually follow a decision-first paradigm, where the model first decides on the action based on past conversations, and then executes the corresponding action. Since LLMs struggle to process a large number of candidate items, the recommendation process is often carried out in collaboration with external recommendation tools, which provide a small candidate set for LLMs to refine.

The past few years have witnessed a growing interest in LLM-based recommender systems (RSs), although their industrial deployment remains in a preliminary stage. Most existing deployments leverage LLMs offline as feature enhancers, generating augmented knowledge for downstream tasks. However, in recommendation scenarios with numerous users and items, even offline knowledge generation with LLMs demands significant time and computational resources. This inefficiency arises from the autoregressive nature of LLMs. A promising solution is speculative decoding, a Draft-Then-Verify approach that increases the number of tokens generated per decoding step. In this work, we first identify recommendation knowledge generation as a highly fitting use case for retrieval-based speculative decoding. Then, we discern its two characteristics: (1) the vast number of items and users in RSs leads to retrieval inefficiency, and (2) RSs exhibit high diversity tolerance for LLM-generated text. Building on these insights, we introduce Lossless Acceleration via Speculative Decoding for LLM-based Recommender Systems (LASER), which features a Customized Retrieval Pool to enhance retrieval efficiency and Relaxed Verification to improve the acceptance rate of draft tokens. LASER achieves a 3-5x speedup on public datasets and saves about 67% of computational resources during the online A/B test on a large-scale advertising scenario with lossless downstream recommendation performance. Our code is available at https://github.com/YunjiaXi/LASER

Reranking is significant for recommender systems due to its pivotal role in refining recommendation results. Numerous reranking models have emerged to meet diverse reranking requirements in practical applications, which not only prioritize accuracy but also consider additional aspects such as diversity and fairness. However, most of the existing models struggle to strike a harmonious balance between these diverse aspects at the model level. Additionally, the scalability and personalization of these models are often limited by their complexity and a lack of attention to the varying importance of different aspects in diverse reranking scenarios. To address these issues, we propose LLM4Rerank, a comprehensive LLM-based reranking framework designed to bridge the gap between various reranking aspects while ensuring scalability and personalized performance. Specifically, we abstract different aspects into distinct nodes and construct a fully connected graph for LLM to automatically consider aspects like accuracy, diversity, fairness, and more, all in a coherent Chain-of-Thought (CoT) process. To further enhance personalization during reranking, we facilitate a customizable input mechanism that allows fine-tuning of LLM's focus on different aspects according to specific reranking needs. Experimental results on three widely used public datasets demonstrate that LLM4Rerank outperforms existing state-of-the-art reranking models across multiple aspects.

A vast amount of user behavior data is constantly accumulating on today's large recommendation platforms, recording users' various interests and tastes. Preserving knowledge from the old data while new data continually arrives is a vital problem for recommender systems. Existing approaches generally seek to save the knowledge implicitly in the model parameters. However, such a parameter-centric approach lacks scalability and flexibility---the capacity is hard to scale, and the knowledge is inflexible to utilize. Hence, in this work, we propose a framework that turns massive user behavior data to retrievable knowledge (D2K). It is a data-centric approach that is model-agnostic and easy to scale up. Different from only storing unary knowledge such as the user-side or item-side information, D2K propose to store ternary knowledge for recommendation, which is determined by the complete recommendation factors---user, item, and context. The knowledge retrieved by target samples can be directly used to enhance the performance of any recommendation algorithms. Specifically, we introduce a Transformer-based knowledge encoder to transform the old data into knowledge with the user-item-context cross features. A personalized knowledge adaptation unit is devised to effectively exploit the information from the knowledge base by adapting the retrieved knowledge to the target samples. Extensive experiments demonstrate the superiority of the proposed method.

With the rapid development of online services and web applications, recommender systems (RS) have become increasingly indispensable for mitigating information overload and matching users’ information needs by providing personalized suggestions over items. Although the RS research community has made remarkable progress over the past decades, conventional recommendation models (CRM) still have some limitations, e.g., lacking open-domain world knowledge, and difficulties in comprehending users’ underlying preferences and motivations. Meanwhile, large language models (LLM) have shown impressive general intelligence and human-like capabilities for various natural language processing (NLP) tasks, which mainly stem from their extensive open-world knowledge, logical and commonsense reasoning abilities, as well as their comprehension of human culture and society. Consequently, the emergence of LLM is inspiring the design of RS and pointing out a promising research direction, i.e., whether we can incorporate LLM and benefit from their common knowledge and capabilities to compensate for the limitations of CRM. In this article, we conduct a comprehensive survey on this research direction, and draw a bird’s-eye view from the perspective of the whole pipeline in real-world RS. Specifically, we summarize existing research works from two orthogonal aspects: where and how to adapt LLM to RS. For the “ WHERE ” question, we discuss the roles that LLM could play in different stages of the recommendation pipeline, i.e., feature engineering, feature encoder, scoring/ranking function, user interaction, and pipeline controller. For the “ HOW ” question, we investigate the training and inference strategies, resulting in two fine-grained taxonomy criteria, i.e., whether to tune LLM or not during training, and whether to involve CRM for inference. Detailed analysis and general development paths are provided for both “WHERE” and “HOW” questions, respectively. Then, we highlight the key challenges in adapting LLM to RS from three aspects, i.e., efficiency, effectiveness, and ethics. Finally, we summarize the survey and discuss the future prospects.

Recent studies highlight the potential of large language models (LLMs) to enhance content integration in recommender systems by leveraging their semantic understanding capabilities. However, directly incorporating LLMs into an online inference pipeline significantly increases computation costs for large-scale deployment, posing a practical challenge in balancing their benefits and costs. In this work, we propose the EASE framework, which enriches and aligns semantic feature embeddings using LLMs during the training phase while establishing a lightweight inference pipeline that does not directly involve LLMs. Specifically, we train a semantic adapter to align item features with LLMs and simultaneously enrich semantic embeddings through reconstruction tasks from LLMs. During inference, we retain only the item feature encoder and lightweight semantic adapter, thereby eliminating the computation overhead of resource-intensive LLMs. Our EASE framework is flexible, supporting not only text and visual features but also other pre-processed embedding features. Extensive experiments on both public and industrial datasets demonstrate that enriching semantic feature embeddings with our EASE framework yields consistent improvements in downstream click-through rate prediction tasks.

Conversational recommender systems (CRSs) aim to capture user preferences and provide personalized recommendations through multi-round natural language dialogues. However, most existing CRS models mainly focus on dialogue comprehension and preferences mining from the current dialogue session, overlooking user preferences in historical dialogue sessions. The preferences embedded in historical sessions and the current session exhibit continuity and sequentiality, and we refer to such CRSs as sequential CRSs. In this work, we leverage memory-enhanced LLMs to model the preference continuity, addressing two key issues: (1) redundancy and noise in historical dialogue sessions, and (2) the cold-start users problem. Thus, we propose a Memory-enhanced Conversational Recommender System Framework with Large Language Models (dubbed MemoCRS), consisting of user-specific memory and general memory. User-specific memory is tailored to each user's interests and uses an entity-based memory bank to refine preferences and retrieve relevant memory, thereby reducing the redundancy and noise of historical sessions. The general memory, encapsulating collaborative knowledge and reasoning guidelines, can provide shared knowledge for users, especially cold-start users. With the above memory, LLMs are empowered to deliver more precise and tailored recommendations for each user. Extensive experiments on Chinese and English datasets demonstrate MemoCRS's effectiveness.

Recommender system plays a vital role in various online services. However, its insulated nature of training and deploying separately within a specific closed domain limits its access to open-world knowledge. Recently, the emergence of large language models (LLMs) has shown promise in bridging this gap by encoding extensive world knowledge and demonstrating reasoning capabilities. Nevertheless, previous attempts to directly use LLMs as recommenders cannot meet the inference latency demand of industrial recommender systems. In this work, we propose an Open-World Knowledge Augmented Recommendation Framework with Large Language Models, dubbed KAR, to acquire two types of external knowledge from LLMs — the reasoning knowledge on user preferences and the factual knowledge on items. We introduce factorization prompting to elicit accurate reasoning on user preferences. The generated reasoning and factual knowledge are effectively transformed and condensed into augmented vectors by a hybrid-expert adaptor in order to be compatible with the recommendation task. The obtained vectors can then be directly used to enhance the performance of any recommendation model. We also ensure efficient inference by preprocessing and prestoring the knowledge from the LLM. Extensive experiments show that KAR significantly outperforms the state-of-the-art baselines and is compatible with a wide range of recommendation algorithms. We deploy KAR to Huawei’s news and music recommendation platforms and gain a 7% and 1.7% improvement in the online A/B test, respectively.

Real-Time Bidding (RTB) is one of the most important forms of online advertising, where an auction is hosted in real time to sell the individual ad impression. How to design an automated bidding strategy in response to the dynamic auction environment is crucial for improving user experience, protecting the interests of advertisers, and promoting the long-term development of the advertising platform. As an exciting topic in the real-world industry, it has attracted great research interest from several disciplines, most notably data science. There have been abundant studies on bidding strategy design which are based on the large volume of historical ad requests. Despite its popularity and significance, few works provide a summary for bid optimization. In this survey, we present the latest overview of the recent works to shed light on the optimization techniques where most of them are validated in practice. We first explore the optimization problem in different works, explaining how these different settings affect the bidding strategy designs. Then, some forms of bidding functions and specific optimization techniques are illustrated. Further, we specifically discuss a new trend about bidding in first-price auctions, which have gradually become popular in recent years. From this survey, both practitioners and researchers can gain insights of the challenges and future prospects of bid optimization in RTB.

As a critical task for large-scale commercial recommender systems, reranking rearranges items in the initial ranking lists from the previous ranking stage to better meet users’ demands. Foundational work in reranking has shown the potential of improving recommendation results by uncovering mutual influence among items. However, rather than considering the context of initial lists as most existing methods do, an ideal reranking algorithm should consider the counterfactual context— the position and the alignment of the items in the reranked lists . In this work, we propose a novel pairwise reranking framework, Utility-oriented Reranking with Counterfactual Context (URCC), which maximizes the overall utility after reranking efficiently. Specifically, we first design a utility-oriented evaluator, which applies Bi-LSTM and graph attention mechanism to estimate the listwise utility via the counterfactual context modeling. Then, under the guidance of the evaluator, we propose a pairwise reranker model to find the most suitable position for each item by swapping misplaced item pairs. Extensive experiments on two benchmark datasets and a proprietary real-world dataset demonstrate that URCC significantly outperforms the state-of-the-art models in terms of both relevance-based metrics and utility-based metrics.

CTR prediction is critical to industrial recommender systems. Recently, with the growth of business domains in enterprises, much attention has been focused on the multi-domain CTR recommendation. Numerous models have been proposed that attempt to use a unified model to serve multiple domains. Although much progress has been made, we argue that they ignore the importance of feature interactions and user behaviors when modeling cross-domain relations, which is a coarse-grained utilizing of domain information. To solve this problem, we propose Domain Facilitated Feature Modeling (DFFM) for CTR prediction. It incorporates domain-related information into the parameters of the feature interaction and user behavior modules, allowing for domain-specific learning of these two aspects. Extensive experiments are conducted on two public datasets and one industrial dataset to demonstrate the effectiveness of DFFM. We deploy the DFFM model in Huawei advertising platform and gain a 4.13% improvement of revenue on a two week online A/B test. Currently DFFM model has been used as the main traffic model, serving for hundreds of millions of people.

Multi-domain recommendation (MDR) aims to provide recommendations for different domains (e.g., types of products) with overlapping users/items and is common for platforms such as Amazon, Facebook, and LinkedIn that host multiple services. Existing MDR models face two challenges: First, it is difficult to disentangle knowledge that generalizes across domains (e.g., a user likes cheap items) and knowledge specific to a single domain (e.g., a user likes blue clothing but not blue cars). Second, they have limited ability to transfer knowledge across domains with small overlaps. We propose a new MDR method named EDDA with two key components, i.e., embedding disentangling recommender and domain alignment, to tackle the two challenges respectively. In particular, the embedding disentangling recommender separates both the model and embedding for the inter-domain part and the intra-domain part, while most existing MDR methods only focus on model-level disentangling. The domain alignment leverages random walks from graph processing to identify similar user/item pairs from different domains and encourages similar user/item pairs to have similar embeddings, enhancing knowledge transfer. We compare EDDA with 12 state-of-the-art baselines on 3 real datasets. The results show that EDDA consistently outperforms the baselines on all datasets and domains. All datasets and codes are available at https://github.com/Stevenn9981/EDDA.

Position auctions are widely studied in the context of sponsored search advertising, where multiple ad slots are sold in a single auction. In traditional sponsored search, bids are submitted at the keyword level, while recent works have explored transitioning to impression-level bidding using Real-Time Bidding (RTB) techniques to achieve finer bidding. However, position auctions introduce varying user appeal across different positions and more dynamic auction landscape, which RTB, originally devised for single-slot display advertising, fails to address adequately.

The boom of generative models has paved the way for significant advances in recommender systems. For instance, pre-trained generative models offer unprecedented opportunities to improve recommender algorithms for user modeling. This workshop aims to provide a platform for researchers to actively explore and share innovative ideas on integrating generative models into recommender systems, mainly focusing on five key aspects: (i) enhancing recommender algorithms, (ii) generating personalized content in some scenarios such as micro-videos, (iii) changes in the user-system interaction paradigm, (iv) boosting trustworthiness checks, and (v) evaluation methodologies of generative recommendation. With the rapid development of generative models, a growing number of studies along the above directions are emerging, revealing the timeliness and necessity of this workshop. The related research will bring novel features to recommender systems and contribute to new tasks and technologies in both academia and industry. In the long run, this research direction might revolutionize the traditional recommender paradigm and lead to the maturation of next-generation recommender systems.

Re-ranking, as the final stage of the multi-stage recommender systems (MRS), aims at modeling the listwise context and the cross-item interactions between the candidate items. The objective is usually the overall utility (e.g., total clicks or revenue) of the re-ranked list, which is determined not only by the relevance, but also by the diversity of the list. However, existing methods equally promote diversity for all users and often compromise the relevance ranking. In reality, users have different diversity preferences and we should diversify the list tailored to individual users’ interests and needs. Users’ behavior history contains rich information which may be used for inferring their diversity preferences, but has rarely been explored in existing work. In this work, we propose a novel neural re-ranking with personalized diversification method (dubbed RAPID) to address the above challenge. RAPID explicitly models each user’s preference distribution over different topics by exploiting the intra- and inter-topic interactions from the user’s behavior history. The personalized diversity gain brought by each candidate item is then measured by the item’s marginal diversity and the learned personalized preference. The relevance and the personalized diversity are jointly optimized in an end-to-end manner to automatically manage the relevance-diversity tradeoff. Experimental results on two public datasets and a proprietary dataset show that RAPID outperforms the state-of-the-art with the highest utility and the best relevance-diversity tradeoff. We further prove that RAPID has a regret bound of $\tilde O(\sqrt n )$ on utility, which provides theoretical guarantee that its performance is near-optimal.

Real-Time Bidding (RTB) has shown remarkable success in display advertising and has been employed in other advertising scenarios, e.g., sponsored search advertising with multiple ad slots. Many current RTB techniques built for single-slot display advertising are thus no longer applicable, especially in the bid landscape forecasting. Landscape forecasting predicts market competition, including the highest bid price and winning probability, which is preliminary and crucial for the subsequent bidding strategy design. In the multi-slot advertising, predicting the winning prices for each position requires a more precise differentiation of bids among top advertisers. Furthermore, defining the winning probability and addressing censorship issues are not as straightforward as in the case of a single slot. In view of these challenges, how to forecast the bidding landscape in the multi-slot environment remains open.

As an emerging field driven by industrial applications, integrated re-ranking combines lists from upstream sources into a single list, and presents it to the user. The quality of integrated re-ranking is especially sensitive to real-time user behaviors and preferences. However, existing methods are all built on the cloud-to-edge framework, where mixed lists are generated by the cloud model and then sent to the devices. Despite its effectiveness, such a framework fails to capture users' real-time preferences due to the network bandwidth and latency. Hence, we propose to place the integrated re-ranking model on devices, allowing for the full exploitation of real-time behaviors. To achieve this, we need to address two key issues: first, how to extract users' preferences for different sources from heterogeneous and imbalanced user behaviors; second, how to explore the correlation between the extracted personalized preferences and the candidate items. In this work, we present the first on-Device Integrated Re-ranking framework, DIR, to avoid delays in processing real-time user behaviors. DIR includes a multi-sequence behavior modeling module to extract the user's source-level preferences, and a preference-adaptive re-ranking module to incorporate personalized source-level preferences into the re-ranking of candidate items. Besides, we design exposure loss and utility loss to jointly optimize exposure fairness and overall utility. Extensive experiments on three datasets show that DIR significantly outperforms the state-of-the-art baselines in utility-based and fairness-based metrics.

User Behavior Modeling (UBM) plays a critical role in user interest learning, and has been extensively used in recommender systems. The exploration of key interactive patterns between users and items has yielded significant improvements and great commercial success across a variety of recommendation tasks. This tutorial aims to offer an in-depth exploration of this evolving research topic. We start by reviewing the research background of UBM, paving the way to a clearer understanding of the opportunities and challenges. Then, we present a systematic categorization of existing UBM research works, which can be categorized into four different directions including Conventional UBM, Long-Sequence UBM, Multi-Type UBM, and UBM with Side Information. To provide an expansive understanding, we delve into each category, discussing representative models while highlighting their respective strengths and weaknesses. Furthermore, we elucidate on the industrial applications of UBM methods, aiming to provide insights into the practical value of existing UBM solutions. Finally, we identify some open challenges and future prospects in UBM. This comprehensive tutorial serves to provide a solid foundation for anyone looking to understand and implement UBM in their research or business.

extended-abstract Share on International Workshop on Deep Learning Practice for High-Dimensional Sparse Data with RecSys 2023 Authors: Ruiming Tang Huawei Noah's Ark Lab, China Huawei Noah's Ark Lab, China 0000-0002-9224-2431View Profile , Xiaoqiang Zhu Mobvista Group, China Mobvista Group, China 0000-0001-7486-0853View Profile , Junfeng Ge Alibaba Group, China Alibaba Group, China 0000-0001-8435-0443View Profile , Kuang-chih Lee Alibaba Group, USA Alibaba Group, USA 0009-0007-5198-9866View Profile , Biye Jiang Alibaba Group, China Alibaba Group, China 0009-0001-5814-1581View Profile , Xingxing Wang Meituan, China Meituan, China 0000-0002-2655-3928View Profile , Han Zhu Alibaba Group, China Alibaba Group, China 0000-0002-9522-5637View Profile , Tao Zhuang Alibaba Group, China Alibaba Group, China 0000-0002-7408-8514View Profile , Weiwen Liu Huawei Noah's Ark Lab, China Huawei Noah's Ark Lab, China 0000-0002-9148-3997View Profile , Kan Ren Microsoft Research, China Microsoft Research, China 0000-0002-4032-9615View Profile , Weinan Zhang Shanghai Jiao Tong University, China Shanghai Jiao Tong University, China 0000-0002-0127-2425View Profile , Xiangyu Zhao City University of Hong Kong, China City University of Hong Kong, China 0000-0003-2926-4416View Profile Authors Info & Claims RecSys '23: Proceedings of the 17th ACM Conference on Recommender SystemsSeptember 2023Pages 1276–1280https://doi.org/10.1145/3604915.3608765Published:14 September 2023Publication History 0citation67DownloadsMetricsTotal Citations0Total Downloads67Last 12 Months67Last 6 weeks67 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Most click models focus on user behaviors towards a single list. However, with the development of user interface (UI) design, the layout of displayed items on a result page tends to be multi-block style instead of a single list, which requires different assumptions to model user behaviors more accurately. There exist click models for multi-block pages in desktop contexts, but they cannot be directly applied to mobile scenarios due to different interaction manners, result types and especially multi-block presentation styles. In particular, multi-block mobile pages can normally be decomposed into interleavings of basic vertical blocks and horizontal blocks, thus resulting in typically F-shape forms. To mitigate gaps between desktop and mobile contexts for multi-block pages, we conduct a user eye-tracking study, and identify users' sequential browsing, block skip and comparison patterns on F-shape pages. These findings lead to the design of a novel F-shape Click Model (FSCM), which serves as a general solution to multi-block mobile pages. Firstly, we construct a Directed Acyclic Graph (DAG) for each page, where each item is regarded as a vertex and each edge indicates the user's possible examination flow. Secondly, we propose DAG-structured GRUs and a comparison module to model users' sequential (sequential browsing, block skip) and non-sequential (comparison) behaviors respectively. Finally, we combine GRU states and comparison patterns to perform user click predictions. Experiments show that FSCM outperforms baseline models.

Reranking, as the final stage of multi-stage recommender systems, refines the initial lists to maximize the total utility. With the development of multimedia and user interface design, the recommendation page has evolved to a multi-list style. Separately employing traditional list-level reranking methods for different lists overlooks the inter-list interactions and the effect of different page formats, thus yielding suboptimal reranking performance. Moreover, simply applying a shared network for all the lists fails to capture the commonalities and distinctions in user behaviors on different lists. To this end, we propose to draw a bird's-eye view of page-level reranking and design a novel Page-level Attentional Reranking (PAR) model. We introduce a hierarchical dual-side attention module to extract personalized intra- and inter-list interactions. A spatial-scaled attention network is devised to integrate the spatial relationship into pairwise item influences, which explicitly models the page format. The multi-gated mixture-of-experts module is further applied to capture the commonalities and differences of user behaviors between different lists. Extensive experiments on a public dataset and a proprietary dataset show that PAR significantly outperforms existing baseline models.

Click-through rate (CTR) estimation plays a crucial role in modern online personalization services. It is essential to capture users’ drifting interests by modeling sequential user behaviors to build an accurate CTR estimation model. However, as the users accumulate a large amount of behavioral data on the online platforms, the current CTR models have to truncate user behavior sequences and utilize the most recent behaviors, which leads to a problem that sequential patterns such as periodicity or long-term dependency are not contained in the recent behaviors but in far back history. However, it is non-trivial to model the entire user sequence by directly using it for two reasons. Firstly, the very long input sequences will make online inference time and system load infeasible. Secondly, the very long sequences contain much noise, thus making it difficult for CTR models to capture useful patterns effectively. To tackle this issue, we consider it from the input data perspective instead of designing more sophisticated yet complex models. As the entire user behavior sequence contains much noise, it is unnecessary to input the entire sequence. Instead, we could just retrieve only a small part of it as the input to the CTR model. In this article, we propose the U ser B ehavior R etrieval (UBR) framework which aims at learning to retrieve the most informative user behaviors according to each CTR estimation request. Retrieving only a small set of behaviors could alleviate the two problems of utilizing very long sequences (i.e., inference efficiency and noisy input). The distinguishing property of UBR is that it supports arbitrary and learnable retrieval functions instead of utilizing a fixed pre-defined function, which is different from the current retrieval-based methods. Offline evaluations on three large-scale real-world datasets demonstrate the superiority and efficacy of the UBR framework. We further deploy UBR at the Huawei App Store, where it achieves 6.6% of eCPM gain in the online A/B test and now serves the main traffic in the Huawei App Store advertising scenario.

Numerical features (e.g., statistical features) are widely used in recommender systems and online advertising. Existing approaches for numerical feature representation in industry are primarily based on discretization. However, hard-discretization based methods (e.g., Equal Distance Discretization) are deficient in continuity while soft-discretization based methods (e.g., AutoDis) lack discriminability. To emphasize both continuity and discriminability for numerical features, we propose an end-to-end representation learning framework named NaryDis. Specifically, NaryDis first leverages hybrid n-ary encoding as an automatic discretization module to generate hybrid-grained discretization results (multiple encoded sequences). Each position of the encoded sequence is assigned with a positional embedding and an intra-ary attention network is leveraged to aggregate the positional embeddings for obtaining ary-wise representations. Then an inter-ary attention is adopted to assemble these representations, which are further constrained by a self-supervised regularization module. Comprehensive experiments on two public datasets are conducted to show the superiority and compatibility of NaryDis. Besides, we deeply investigate the properties of continuity and discriminability. Moreover, we further verify the effectiveness of NaryDis on a large-scale industrial advertisement dataset.

Integrated ranking is critical in industrial recommendation systems and has attracted increasing attention. In an integrated ranking system, items from multiple channels are merged together and form an integrated list. During this process, apart from optimizing the system's utility like the total number of clicks, a fair allocation of the exposure opportunities over different channels also needs to be satisfied. To address this problem, we propose an integrated ranking model called Integrated Deep-Q Network (iDQN), which jointly considers user preferences, the platform's utility, and the exposure fairness. Extensive offline experiments validate the effectiveness of iDQN in managing the tradeoff between utility and fairness. Moreover, iDQN also has been deployed onto the online AppStore platform in Huawei, where the online A/B test shows iDQN outperforms the baseline by 1.87% and 2.21% in terms of utility and fairness, respectively.

Building appropriate scenarios to meet the personalized demands of different user groups is a common practice. Despite various scenario brings personalized service, it also leads to challenges for the recommendation on multiple scenarios, especially the scenarios with limited traffic. To give desirable recommendation service for all scenarios and reduce the cost of resource consumption, how to leverage the information from multiple scenarios to construct a unified model becomes critical. Unfortunately, the performance of existing multi-scenario recommendation approaches is poor since they introduce unnecessary information from other scenarios to target scenario. In this paper, we show it is possible to selectively utilize the information from different scenarios to construct the scenario-aware estimators in a unified model. Specifically, we first do analysis on multi-scenario modeling with causal graph from the perspective of users and modeling processes, and then propose the Causal Inspired Intervention (CausalInt) framework for multi-scenario recommendation. CausalInt consists of three modules: (1) Invariant Representation Modeling module to squeeze out the scenario-aware information through disentangled representation learning and obtain a scenario-invariant representation; (2) Negative Effects Mitigating module to resolve conflicts between different scenarios and conflicts between scenario-specific and scenario-invariant representations via gradient based orthogonal regularization and model-agnostic meta learning, respectively; (3) Inter-Scenario Transferring module designs a novel TransNet to simulate a counterfactual intervention and effectively fuse the information from other scenarios. Offline experiments over two real-world dataset and online A/B test are conducted to demonstrate the superiority of CausalInt.

Re-ranking is one of the most critical stages for multi-stage recommender systems (MRS), which re-orders the input ranking lists by modeling the cross-item interaction. Recent re-ranking methods have evolved into deep neural architectures due to the significant advances in deep learning. Neural re-ranking, therefore, has become a trending topic and many of the improved algorithms have demonstrated their use in industrial applications, enjoying great commercial success. The purpose of this tutorial is to explore some of the recent work on neural re-ranking, integrating them into a broader picture and paving ways for more comprehensive solutions for future research. In particular, we provide a taxonomy of current methods according to the objectives and training signals. We examine and compare these methods qualitatively and quantitatively, and identify some open challenges and future prospects.

Building a multi-stage cascade ranking system is a commonly used solution to balance the efficiency and effectiveness in modern information retrieval (IR) applications, such as recommendation and web search. Despite the popularity in practice, the literature specific on multi-stage cascade ranking systems is relatively scarce. The common practice is to train rankers of each stage independently using the same user feedback data (a.k.a., impression data), disregarding the data flow and the possible interactions between stages. This straightforward solution could lead to a sub-optimal system because of the sample selection bias (SSB) issue, which is especially damaging for cascade rankers due to the negative effect accumulated in the multiple stages. Worse still, the interactions between the rankers of each stage are not fully exploited. This paper provides an elaborate analysis of this commonly used solution to reveal its limitations. By studying the essence of cascade ranking, we propose a joint training framework named RankFlow to alleviate the SSB issue and exploit the interactions between the cascade rankers, which is the first systematic solution for this topic. We propose a paradigm of training cascade rankers that emphasizes the importance of fitting rankers on stage-specific data distributions instead of the unified user feedback distribution. We design the RankFlow framework based on this paradigm: The training data of each stage is generated by its preceding stages while the guidance signals not only come from the logs but its successors. Extensive experiments are conducted on various IR scenarios, including recommendation, web search and advertisement. The results verify the efficacy and superiority of RankFlow.

As the final stage of the multi-stage recommender system (MRS), reranking directly affects users' experience and satisfaction, thus playing a critical role in MRS. Despite the improvement achieved in the existing work, three issues are yet to be solved. First, users' historical behaviors contain rich preference information, such as users' long and short-term interests, but are not fully exploited in reranking. Previous work typically treats items in history equally important, neglecting the dynamic interaction between the history and candidate items. Second, existing reranking models focus on learning interactions at the item level while ignoring the fine-grained feature-level interactions. Lastly, estimating the reranking score on the ordered initial list before reranking may lead to the early scoring problem, thereby yielding suboptimal reranking performance. To address the above issues, we propose a framework named Multi-level Interaction Reranking (MIR). MIR combines low-level cross-item interaction and high-level set-to-list interaction, where we view the candidate items to be reranked as a set and the users' behavior history in chronological order as a list. We design a novel SLAttention structure for modeling the set-to-list interactions with personalized long-short term interests. Moreover, feature-level interactions are incorporated to capture the fine-grained influence among items. We design MIR in such a way that any permutation of the input items would not change the output ranking, and we theoretically prove it. Extensive experiments on three public and proprietary datasets show that MIR significantly outperforms the state-of-the-art models using various ranking and utility metrics.

Recommender systems are complex. They integrate the individual needs of users with the characteristics of particular domains of application which may span items from large and potentially heterogeneous collections. Extensive experimentation is required to understand the multidimensional properties of recommendation algorithms and the fit between algorithm and application. librec-auto is a tool that automates many aspects of off-line batch recommender system experimentation. It has a large library of state-of-the-art and historical recommendation algorithms and a wide variety of evaluation metrics. It further supports the study of diversity and fairness in recommendation through the integration of re-ranking algorithms and fairness-aware metrics. It supports declarative configuration for reproducible experiment management and supports multiple forms of hyper-parameter optimization.

Prediction over tabular data is an essential task in many data science applications such as recommender systems, online advertising, medical treatment, etc. Tabular data is structured into rows and columns, with each row as a data sample and each column as a feature attribute. Both the columns and rows of the tabular data carry useful patterns that could improve the model prediction performance. However, most existing models focus on the cross-column patterns yet overlook the cross-rowpatterns as they deal with single samples independently. In this work, we propose a general learning framework named Retrieval & Interaction Machine (RIM) that fully exploits both cross-row and cross-column patterns among tabular data. Specifically, RIM first leverages search engine techniques to efficiently retrieve useful rows of the table to assist the label prediction of the target row, then uses feature interaction networks to capture the cross-column patterns among the target row and the retrieved rows so as to make the final label prediction. We conduct extensive experiments on 11 datasets of three important tasks, i.e., CTR prediction (classification), top-n recommendation (ranking) and rating prediction (regression). Experimental results show that RIM achieves significant improvements over the state-of-the-art and various baselines, demonstrating the superiority and efficacy of RIM.

To better exploit search logs and model users' behavior patterns, numerous click models are proposed to extract users' implicit interaction feedback. Most traditional click models are based on the probabilistic graphical model (PGM) framework, which requires manually designed dependencies and may oversimplify user behaviors. Recently, methods based on neural networks are proposed to improve the prediction accuracy of user behaviors by enhancing the expressive ability and allowing flexible dependencies. However, they still suffer from the data sparsity and cold-start problems. In this paper, we propose a novel graph-enhanced click model (GraphCM) for web search. Firstly, we regard each query or document as a vertex, and propose novel homogeneous graph construction methods for queries and documents respectively, to fully exploit both intra-session and inter-session information for the sparsity and cold-start problems. Secondly, following the examination hypothesis, we separately model the attractiveness estimator and examination predictor to output the attractiveness scores and examination probabilities, where graph neural networks and neighbor interaction techniques are applied to extract the auxiliary information encoded in the pre-constructed homogeneous graphs. Finally, we apply combination functions to integrate examination probabilities and attractiveness scores into click predictions. Extensive experiments conducted on three real-world session datasets show that GraphCM not only outperforms the state-of-art models, but also achieves superior performance in addressing the data sparsity and cold-start problems.

Modern information retrieval systems, including web search, ads placement, and recommender systems, typically rely on learning from user feedback. Click models, which study how users interact with a ranked list of items, provide a useful understanding of user feedback for learning ranking models. Constructing ”right” dependencies is the key of any successful click model. However, probabilistic graphical models (PGMs) have to rely on manually assigned dependencies, and oversimplify user behaviors. Existing neural network based methods promote PGMs by enhancing the expressive ability and allowing flexible dependencies, but still suffer from exposure bias and inferior estimation. In this paper, we propose a novel framework, Adversarial Imitation Click Model (AICM), based on imitation learning. Firstly, we explicitly learn the reward function that recovers users’ intrinsic utility and underlying intentions. Secondly, we model user interactions with a ranked list as a dynamic system instead of one-step click prediction, alleviating the exposure bias problem. Finally, we minimize the JS divergence through adversarial training and learn a stable distribution of click sequences, which makes AICM generalize well across different distributions of ranked lists. A theoretical analysis has indicated that AICM reduces the exposure bias from O(T2) to O(T). Our studies on a public web search dataset show that AICM not only outperforms state-of-the-art models in traditional click metrics but also achieves superior performance in addressing the exposure bias and recovering the underlying patterns of click sequences.

Re-ranking is a critical task for large-scale commercial recommender systems. Given the initial ranked lists, top candidates are re-ranked to improve the accuracy of the ranking results. However, existing re-ranking strategies are sub-optimal due to (i) most prior works do not consider explicit item relationships, like being substitutable or complementary, which may mutually influence the user satisfaction on other items in the lists, and (ii) they usually apply an identical re-ranking strategy for all users, with personalized user preferences and intents ignored. To resolve the problem, we construct a heterogeneous graph to fuse the initial scoring information and item relationships information. We develop a graph neural network based framework, IRGPR, to explicitly model transitive item relationships by recursively aggregating relational information from multi-hop neighborhoods. We also incorporate a novel intent embedding network to embed personalized user intents into the propagation. We conduct extensive experiments on real-world datasets, demonstrating the effectiveness of IRGPR in re-ranking. Further analysis reveals that modeling the item relationships and personalized intents are particularly useful for improving the performance of re-ranking.

User-generated item lists are popular on many platforms. Examples include video-based playlists on YouTube, image-based lists (or "boards") on Pinterest, book-based lists on Goodreads, and answer-based lists on question-answer forums like Zhihu. As users create these lists, a common challenge is in identifying what items to curate next. Some lists are organized around particular genres or topics, while others are seemingly incoherent, reflecting individual preferences for what items belong together. Furthermore, this heterogeneity in item consistency may vary from platform to platform, and from sub-community to sub-community. Hence, this paper proposes a generalizable approach for user-generated item list continuation. Complementary to methods that exploit specific content patterns (e.g., as in song-based playlists that rely on audio features), the proposed approach models the consistency of item lists based on human curation patterns, and so can be deployed across a wide range of varying item types (e.g., videos, images, books). A key contribution is in intelligently combining two preference models via a novel consistency-aware gating network -- a general user preference model that captures a user's overall interests, and a current preference priority model that captures a user's current (as of the most recent item) interests. In this way, the proposed consistency-aware recommender can dynamically adapt as user preferences evolve. Evaluation over four datasets (of songs, books, and answers) confirms these observations and demonstrates the effectiveness of the proposed model versus state-of-the-art alternatives. Further, all code and data are available at https://github.com/heyunh2015/ListContinuation_WSDM2020.

Microlending can lead to improved access to capital in impoverished countries. Recommender systems could be used in microlending to provide efficient and personalized service to lenders. However, increasing concerns about discrimination in machine learning hinder the application of recommender systems to the microfinance industry. Most previous recommender systems focus on pure personalization, with fairness issue largely ignored. A desirable fairness property in microlending is to give borrowers from different demographic groups a fair chance of being recommended, as stated by Kiva. To achieve this goal, we propose a Fairness-Aware Re-ranking (FAR) algorithm to balance ranking quality and borrower-side fairness. Furthermore, we take into consideration that lenders may differ in their receptivity to the diversification of recommended loans, and develop a Personalized Fairness-Aware Re-ranking (PFAR) algorithm. Experiments on a real-world dataset from Kiva.org show that our re-ranking algorithm can significantly promote fairness with little sacrifice in accuracy, and be attentive to individual lender preference on loan diversity.

For Click-Through Rate (CTR) prediction, Field-aware Factorization Machines (FFM) have exhibited great effectiveness by considering field information. However, it is also observed that FFM suffers from the overfitting problem in many practical scenarios. In this paper, we propose a Field-aware Probabilistic Embedding Neural Network (FPENN) model with both good generalization ability and high accuracy. FPENN estimates the probability distribution of the field-aware embedding rather than using the single point estimation (the maximum a posteriori estimation) to prevent overfitting. Both low-order and high-order feature interactions are considered to improve the accuracy. FPENN consists of three components, i.e., FPE component, Quadratic component and Deep component. FPE component outputs probabilistic embedding to the other two components, where various confidence levels for feature embeddings are incorporated to enhance the robustness and the accuracy. Quadratic component is designed for extracting low-order feature interactions, while Deep component aims at capturing high-order feature interactions. Experiments are conducted on two benchmark datasets, Avazu and Criteo. The results confirm that our model alleviates the overfitting problem while having a higher accuracy.