In this work, we propose a novel Hierarchical Information Sharing and Transfer (HIST) framework for multi-task learning, which employs implicit shared-bottom pattern and explicit sequential transfer at tower-level simultaneously. In particular, a multi-level gating mixture-of-experts is presented for efficient bottom-level sharing. Further, self-attention mechanism is adopted for information transfer between task-specific towers. Such hierarchical task interaction scheme leads to a remarkable enhancement in multi-task learning settings. Extensive experiments on four subsets of AliExpress dataset unequivocally demonstrate that HIST outperforms the current state-of-the-art methods consistently.

Generative retrieval (GR) has revolutionized document retrieval with the advent of large language models (LLMs), and LLM-based GR is gradually being adopted by the industry. Despite its remarkable advantages and potential, LLM-based GR suffers from hallucination and generates documents that are irrelevant to the query in some instances, severely challenging its credibility in practical applications. We thereby propose an optimized GR framework designed to alleviate retrieval hallucination, which integrates knowledge distillation reasoning in model training and incorporate decision agent to further improve retrieval precision. Specifically, we employ LLMs to assess and reason GR retrieved query-document (q-d) pairs, and then distill the reasoning data as transferred knowledge to the GR model. Moreover, we utilize a decision agent as post-processing to extend the GR retrieved documents through retrieval model and select the most relevant ones from multi perspectives as the final generative retrieval result. Extensive offline experiments on real-world datasets and online A/B tests on Fund Search and Insurance Search in Alipay demonstrate our framework's superiority and effectiveness in improving search quality and conversion gains.

Pre-ranking plays a crucial role in large-scale recommender systems by significantly improving the efficiency and scalability within the constraints of providing high-quality candidate sets in real time. The two-tower model is widely used in pre-ranking systems due to a good balance between efficiency and effectiveness with decoupled architecture, which independently processes user and item inputs before calculating their interaction (e.g. dot product or similarity measure). However, this independence also leads to the lack of information interaction between the two towers, resulting in less effectiveness. In this paper, a novel architecture named learnable Fully Interacted Two-tower Model (FIT) is proposed, which enables rich information interactions while ensuring inference efficiency. FIT mainly consists of two parts: Meta Query Module (MQM) and Lightweight Similarity Scorer (LSS). Specifically, MQM introduces a learnable item meta matrix to achieve expressive early interaction between user and item features. Moreover, LSS is designed to further obtain effective late interaction between the user and item towers. Finally, experimental results on several public datasets show that our proposed FIT significantly outperforms the state-of-the-art baseline pre-ranking models.

Query rewriting is essential in e-commerce search, as it bridges the lexical gap between user queries and item descriptions, thereby enhancing search performance.Despite recent advancements, current rewriting approaches are still limited by an inadequate comprehension of domain-specific knowledge and a lack of mechanisms for adaptive refinement in response to new or changing queryitem relationships.To overcome these limitations, we propose a large language model (LLM) based Multi-Agent Framework for Adaptive Query Rewriting (MAAQR) in Alipay Search.Initially, we perform knowledge-enhanced fine-tuning to improve the LLM's understanding of query and item semantics.Subsequently, a multiagent collaborative rewriting architecture is employed to enhance rewrite quality and adaptability.MAAQR has been successfully deployed to serve Alipay's mini-app search since December 2024.Through offline experiments and online A/B testing, MAAQR significantly improves click-through rates (CTR) and the number of transactions for target queries, while substantially reducing the zero-results rate (ZRR).

In recent years, there has been a trend in the field of recommender systems towards multi-task modeling and multi-scenario modeling. The aim is to enhance the performance of various tasks and scenarios by jointly training on multiple tasks or scenarios to learn common patterns and features. Joint modeling of tasks and scenarios has also received widespread attention recently. However, despite the rich proposals of methods for Multi-Task Learning (MTL), Multi-Scenario Learning (MSL), and Multi-Task-Multi-Scenario Learning (MTMSL) in recent years, there still lacks a comprehensive benchmark to evaluate these methods. Previous studies often employed different datasets, data processing techniques, data partitioning strategies, and hyperparameter settings, making replication of existing research and fair comparison of experimental results challenging. To address this challenge, we introduce MMLRec, the first unified comprehensive benchmark for evaluating MTL, MSL and MTMSL, featuring consistent dataset processing and identical parameter settings. This benchmark implements a range of MTL, MSL, and MTMSL algorithms, and evaluates them on multiple commonly used recommender systems datasets. Through fair comparative experiments, we find that some structurally simplistic recommendation algorithms are underestimated, as they can achieve comparable results to more complex algorithms while maintaining lower complexity. Furthermore, our experimental analysis indicates that more complex methods exhibit better robustness when there are significant differences between tasks or scenarios. By providing a unified framework (MMLRec), our goal is to promote rapid evaluation and inspire innovative research in this continuously evolving field. We hope that our open-source benchmark can facilitate swift, equitable evaluations, while also fostering further breakthrough research in the domains of MTL, MSL, and MTMSL.

With the rise of large language models (LLMs), recent works have leveraged LLMs to improve the performance of click-through rate (CTR) prediction. However, we argue that a critical obstacle remains in deploying LLMs for practical use: the efficiency of LLMs when processing long textual user behaviors. As user sequences grow longer, the current efficiency of LLMs is inadequate for training on billions of users and items. To break through the efficiency barrier of LLMs, we propose Behavior Aggregated Hierarchical Encoding (BAHE) to enhance the efficiency of LLM-based CTR modeling. Specifically, BAHE proposes a novel hierarchical architecture that decouples the encoding of user behaviors from inter-behavior interactions. Firstly, to prevent computational redundancy from repeated encoding of identical user behaviors, BAHE employs the LLM's pre-trained shallow layers to extract embeddings of the most granular, atomic user behaviors from extensive user sequences and stores them in the offline database. Subsequently, the deeper, trainable layers of the LLM facilitate intricate inter-behavior interactions, thereby generating comprehensive user embeddings. This separation allows the learning of high-level user representations to be independent of low-level behavior encoding, significantly reducing computational complexity. Finally, these refined user embeddings, in conjunction with correspondingly processed item embeddings, are incorporated into the CTR model to compute the CTR scores. Extensive experimental results show that BAHE reduces training time and memory by five times for CTR models using LLMs, especially with longer user sequences. BAHE has been deployed in a real-world system, allowing for daily updates of 50 million CTR data on 8 A100 GPUs, making LLMs practical for industrial CTR prediction.

Click-through rate (CTR) prediction plays a crucial role in recommendation systems, with significant impact on user experience and platform revenue generation. Despite the various public CTR datasets available due to increasing interest from both academia and industry, these datasets have limitations. They cover a limited range of scenarios and predominantly focus on ID-based features, neglecting the vital role of multi-modal features for effective multi-scenario CTR prediction. Moreover, their scale is modest compared to real-world industrial datasets, hindering robust and comprehensive evaluation of complex models. To address these challenges, we introduce a large-scale Multi-Scenario Multi-Modal CTR dataset named AntM2 C, built from real industrial data from Alipay. This dataset offers an impressive breadth and depth of information, covering CTR data from four diverse business scenarios, including advertisements, consumer coupons, mini-programs, and videos. Unlike existing datasets, AntM2 C provides not only ID-based features but also five textual features and one image feature for both users and items, supporting more delicate multi-modal CTR prediction. AntM2 C is also substantially larger than existing datasets, comprising 100 million CTR data. This scale allows for robust and comprehensive evaluation and comparison of CTR prediction models. We employ AntM2 C to construct several typical CTR tasks, including multi-scenario modeling, item and user cold-start modeling, and multi-modal modeling. Initial experiments and comparisons with baseline methods have shown that AntM2 C presents both new challenges and opportunities for CTR models, with the potential to significantly advance CTR research. The AntM2 C dataset is available at https://www.atecup.cn/OfficalDataSet.

Recommender systems are widely used in various online platforms. In the context of sequential recommendation, it is essential to accurately capture the chronological patterns in user activities to generate relevant recommendations. Conventional ID-based sequential recommenders have shown promise but lack comprehensive real-world knowledge about items, limiting their effectiveness. Recent advancements in Large Language Models (LLMs) offer the potential to bridge this gap by leveraging the extensive real-world knowledge encapsulated in LLMs. However, integrating LLMs into sequential recommender systems comes with its own challenges, including inadequate representation of sequential behavior patterns and long inference latency. In this paper, we propose SeRALM (Enhancing Sequential Recommenders with Augmented Knowledge from Aligned Large Language Models) to address these challenges. SeRALM integrates LLMs with conventional ID-based sequential recommenders for sequential recommendation tasks. We combine text-format knowledge generated by LLMs with item IDs and feed this enriched data into ID-based recommenders, benefitting from the strengths of both paradigms. Moreover, we develop a theoretically underpinned alignment training method to refine LLMs' generation using feedback from ID-based recommenders for better knowledge augmentation. We also present an asynchronous technique to expedite the alignment training process. Experimental results on public benchmarks demonstrate that SeRALM significantly improves the performances of ID-based sequential recommenders. Further, a series of ablation studies and analyses corroborate SeRALM's proficiency in steering LLMs to generate more pertinent and advantageous knowledge across diverse scenarios.

Conversion rate (CVR) prediction models play a vital role in recommendation systems. Recent research shows that learning a unified model to serve multiple scenarios is effective for improving overall performance. However, it remains challenging to improve model prediction performance across scenarios at low model parameter cost, and current solutions are hard to robustly model multi-scenario diversity. In this paper, we propose MI-DPG for the multi-scenario CVR prediction, which learns scenario-conditioned dynamic model parameters for each scenario in a more efficient and effective manner. Specifically, we introduce an auxiliary network to generate scenario-conditioned dynamic weighting matrices, which are obtained by combining decomposed scenario-specific and scenario-shared low-rank matrices with parameter efficiency. For each scenario, weighting the backbone model parameters by the weighting matrix helps to specialize the model parameters for different scenarios. It can not only modulate the complete parameter space of the backbone model but also improve the model effectiveness. Furthermore, we design a mutual information regularization to enhance the diversity of model parameters across scenarios by maximizing the mutual information between the scenario-aware input and the scenario-conditioned dynamic weighting matrix. Experiments from three real-world datasets show that MI-DPG outperforms previous multi-scenario recommendation models.

Numerical features are an important type of input for CTR prediction models. Recently, several discretization and numerical transformation methods have been proposed to deal with numerical features. However, existing approaches do not fully consider compatibility with different distributions. Here, we propose a novel numerical feature embedding framework, called Distribution-Aware Embedding (DAE), which is applicable to various numerical feature distributions. First, DAE efficiently approximates the cumulative distribution function by estimating the expectation of the order statistics. Then, the distribution information is applied to the embedding layer by nonlinear interpolation. Finally, to capture both local and global information, we aggregate the embeddings at multiple scales to obtain the final representation. Empirical results validate the effectiveness of DAE compared to the baselines, while demonstrating the adaptability to different CTR models and distributions.

Prediction models for click-through rate (CTR) learn feature interactions underlying user behaviors, which are crucial in recommendation systems. Due to their size and complexity, existing approaches have a limited range of applications. In order to decrease inference delay, knowledge distillation techniques have been used in recommendation systems. Due to the student model's lower capacity, the knowledge distillation process is less effective when there is a significant difference in the complexity of the network architecture between the teacher model and the student model.

The cold-start problem of conversion rate prediction is a common challenge in online advertising systems. To alleviate this problem, a large number of methods either use content information or uncertainty methods, or use meta-learning based methods to improve the ranking performance of cold-start items. However, they can work for cold-start scenarios but fail to adaptively unify warm and cold recommendations into one model, requiring additional human efforts or knowledge to adapt to different scenarios. Meanwhile, none of them pay attention to the discrepancy between model predictions and true likelihoods of cold items, while over- or under-estimation is harmful to the ROI (Return on Investment) of advertising placements. In this paper, in order to address the above issues, we propose a framework called Distribution-Constrained Batch Transformer (DCBT). Specifically, the framework introduces a Transformer module into the batch dimension to automatically choose proper information from warm samples to enhance the representation of cold samples and preserve the property of warm samples. In addition, to avoid the distribution of cold samples being affected by the warm samples, the framework adds MMD loss to constrain the sample distribution before and after feeding into the Transformer module. Extensive offline experiments on two real-world datasets show that our proposed method attains state-of-the-art performance in AUC and PCOC (Predicted CVR over CVR) for cold items and warm items. An online A/B test demonstrates that the DCBT model obtained a 20.08% improvement in CVR and a 13.21% increase in GMV (Gross Merchandise Volume).

A session-based recommendation system (SRS) tries to predict the next possible choice of anonymous users. In recent years, graph neural network (GNN) models have been successfully applied to SRSs and have achieved great success. Using GNN models in SRSs, each session graph is processed successively to obtain the embedding of the node (i.e, each action on an item), which is then imported into the prediction module to generate recommendation results. However, solely depending on the session graph to obtain the node embeddings is not sufficient because each session only involves a few items. Therefore, neighbor sessions have been used to extend the session graph to learn more informative node representations. In this paper, we introduce a S ession-based recommendation MO del based on N eighbor sessions with similar probabilistic int E ntions(SMONE). SMONE models the intentions behind sessions in a probabilistic way and retrieves the neighbor sessions with similar intentions. After the neighbor sessions are found, the target session and its neighbor sessions are modeled as a hypyergraph to learn the contextualized embeddings, which are combined with item embeddings through GNN to produce the final item recommendations. Experiments on real-world datasets prove the effectiveness and superiority of SMONE.

Multi-task learning (MTL) has been successfully applied to a wide range of real-world applications. However, MTL models often suffer from performance degradation with negative transfer due to sharing all features without distinguishing their helpfulness for all tasks. To this end, many works on feature selection for multi-task learning (FS-MTL) have been proposed to alleviate negative transfer between tasks by learning features selectively for each specific task. However, due to latent confounders between features and task targets, the correlations captured by the feature selection modules proposed in these works may fail to reflect the actual effect of the features on the targets. This paper explains negative transfer in FS-MTL from a causal perspective and presents a novel architecture called Causal Feature Selection for Multi-task Learning(CFS-MTL). This method incorporates the idea of causal inference into feature selection for multi-task learning via pseudo-intervention. It aims to select features with more stable causal effects rather than spurious correlations for each task by regularizing the distance between feature ITEs and feature importance. We conduct extensive experiments based on three real-world datasets to demonstrate that our proposed CFS-MTL outperforms state-of-the-art MTL models significantly in the AUC metric.

Conversion Rate (CVR) prediction is a crucial task in online advertising systems. Existing single-domain CVR prediction models suffer from the data sparsity problem since few users purchase items after clicking. In recent years, a robust and effective technique called fine-tuning can transfer knowledge from a data-rich source domain to enhance the CVR prediction performance in a data-sparse target domain. However, since most CVR prediction models have a large number of parameters, fine-tuning all the parameters on a data-sparse domain may lead to over-fitting. In this paper, we propose a general and efficient transfer learning method called Gradient-Flow based Fine-Tuning (GFlow-FT), which only needs to update a subset of parameters (called child network) via pruning the gradients to restrain gradient norm against over-fitting. In addition, our method employs the gradient-flow based measure via calculating the Hessian-gradient product as the criteria for picking the child network, which is superior to the magnitude-based and loss-based measure from empirical results. Extensive experimental results on three real-world datasets from recommendation systems show that GFlow-FT can significantly improve the performance of CVR prediction compared with state-of-the-art fine-tuning approaches.

Multi-task learning (MTL) aims at solving multiple related tasks simultaneously and has experienced rapid growth in recent years. However, MTL models often suffer from performance degeneration with negative transfer due to learning several tasks simultaneously. Some related work attributed the source of the problem is the conflicting gradients. In this case, it is needed to select useful gradient updates for all tasks carefully. To this end, we propose a novel optimization approach for MTL, named GDOD, which manipulates gradients of each task using an orthogonal basis decomposed from the span of all task gradients. GDOD decomposes gradients into task-shared and task-conflict components explicitly and adopts a general update rule for avoiding interference across all task gradients. This allows guiding the update directions depending on the task-shared components. Moreover, we prove the convergence of GDOD theoretically under both convex and non-convex assumptions. Experiment results on several multi-task datasets not only demonstrate the significant improvement of GDOD performed to existing MTL models but also prove that our algorithm outperforms state-of-the-art optimization methods in terms of AUC and Logloss metrics.

Multi-task learning (MTL) has experienced rapid growth in recent years. A typical way of conducting MTL with deep neural networks (DNNs) is either establishing a sort of global feature sharing mechanism across all tasks or assigning each task an individual set of parameters with cross-connections. However, these existing approaches leverage DNNs only to share features of a certain order. Several modelsdemonstrated that explicitly modeling feature sharing with both low-order and high-order features can boost performance. To this end, we propose a model-agnostic sparse routing architecture called MASR, which emphasizes arbitrary order feature sharing for multi-task learning. It is able to choose specific orders of features to route for a given task through learnable latent variables. Moreover, MASR is model-agnostic and can be combined with existing MTL models to share features of both low-order and high-order. Extensive experimental results on several real-world datasets not only confirm the significant improvement of MASR performed to existing MTL models but also outperform existing hybrid architectures in terms of AUC metric.

Click-through rate (CTR) prediction is a long-standing problem in advertising systems. Existing single-domain CTR prediction methods suffer from the data sparsity problem since few users can click advertisements on many items. Recently, cross-domain CTR prediction leverages the relatively richer information from a source domain to improve the performance on a target domain with sparser information, but it cannot explicitly capture users' diverse interests in different domains. In this paper, we propose a novel hierarchical user behavior modeling framework for cross-domain CTR prediction, named HBMNet. HBMNet contains two main components: an element-wise behavior transfer(EWBT) layer and a user representation layer. EWBT layer transfers the information collected from one domain by element-level masks to dynamically highlight the informative elements in another domain. The user representation layer performs behavior-level attention between these behavior representations and the ranking item representation. Extensive experimental results on two cross-domain datasets show that the proposed HBMNet outperforms SOTA models.

In recommender systems, content-based methods and meta-learning involved methods usually have been adopted to alleviate the item cold-start problem. The former consider utilizing item attributes at the feature level and the latter aim at learning a globally shared initialization for all tasks to achieve fast adaptation with limited data at the task level. However, content-based methods only focus on the similarity of item attributes, ignoring the relationships established by user interactions. And for tasks with different distributions, most meta-learning-based methods are difficult to achieve better performance under a single initialization. To address the limitations mentioned above and combine the strengths of both methods, we propose a Task Similarity Aware Meta-Learning (TSAML) framework from two aspects. Specifically, at the feature level, we simultaneously introduce content information and user-item relationships to exploit task similarity. At the task level, we design an automatic soft clustering module to cluster similar tasks and generate the same initialization for similar tasks. Extensive offline experiments demonstrate that the TSAML framework has superior performance and recommends cold items to preferred users more effectively than other state-of-the-art methods.

Recent methods in sequential recommendation focus on learning multi-interest embedding vectors from a user's behavior sequence for the next-item recommendation. However, behavior sequential data may result from users' conformity towards popular items, which entangles users' real interests and tends to recommend popular items by using interest embeddings. In this paper, we propose a novel multi-interest framework with causal embedding for sequential recommendation, called MiceRec. Specifically, we first obtain two embedding layers from behavior sequence by assigning items with separate embeddings for interest and conformity, then extract multiple pure interests from one embedding layer, while the other for users' conformity extraction. According to the colliding effect of causal inference, we mine cause-specific data for training causal embeddings. Our framework significantly outperforms state-of-the-art solutions on two real-world datasets1. We further demonstrate that the learned multi-interest embeddings successfully separate from each other, and show that conformity information is almost squeezed out from interest embeddings.

Click-Through Rate (CTR) prediction plays an important role in many industrial applications, and recently a lot of deep learning methods attempt to improve the performance through feature interaction or user interest models. However, the existing methods fails to explore the potential interest from user behavior sequence effectively. In this paper, we proposed a Non-sequential Interest Interaction Network (NIIN) to tackle this problem. Specifically, we first design the key-vector storage method to construct the sparse representation for any input behavior sequences. Moreover, our NIIN framework introduces a Time Interest Interaction (TII) module to explore interactions among interests. It first includes an efficient fine-grained attention mechanism to score items in behavior sequence for reserving user's diverse interests. After that, our TII module exploits multidimensional partition layer to segment behavior sequence in temporal dimension, and sum pooling operation obtains user's interest in different periods. Furthermore, several fully connect layers model interactions among multiple interests to mine potential interest and give the final score of the target item. Experiments on CTR datasets including Alipay, Taobao and Alimama datasets demonstrate that the NIIN framework outperforms the state-of-the-art methods.

There exists the cold-start problem in the recommendation systems when observed user-item interactions are insufficient. To alleviate this problem, most existing works aim to learn globally shared prior knowledge across all items and be fast adapted to a new item with few interactions. However, such learning techniques are data demanding and work poorly on new items with no interactions. In this applied paper, we present an industrial framework recently deployed on Alipay to address the item cold-start problem in zero-shot scenarios. The proposed framework provides both efficient and high-quality recommendations for cold items with no log data. Specifically, we formulate the cold-start problem as a zero-shot learning problem and build a highly efficient infrastructure to accomplish online zero-shot recommendations used on large-scale platforms. Extensive offline experiments and online A/B testing demonstrate that the proposed framework has superior performance and recommends cold items to preferred users more effectively than other state-of-the-art methods.

Multi-task learning(MTL) is an open and challenging problem in various real-world applications. The typical way of conducting multi-task learning is establishing some global parameter sharing mechanism across all tasks or assigning each task an individual set of parameters with cross-connections between tasks. However, for most existing approaches, all tasks just thoroughly or proportionally share all the features without distinguishing the helpfulness of them. By that, some tasks would be intervened by the unhelpful features that are useful for other tasks, leading to undesired negative transfer between tasks. In this paper, we design a novel architecture named the Multiple-level Sparse Sharing Model (MSSM), which can learn features selectively and share knowledge across all tasks efficiently. MSSM first employs a field-level sparse connection module (FSCM) to enable much more expressive combinations of feature fields to be learned for generalization across tasks while still allowing for task-specific features to be customized for each task. Furthermore, a cell-level sparse sharing module (CSSM) can recognize the sharing pattern through a set of coding variables that selectively choose which cells to route for a given task. Extensive experimental results on several real-world datasets show that MSSM outperforms SOTA models significantly in terms of AUC and LogLoss metrics.

Transfer learning leverages knowledge from a source domain with rich data to a target domain with sparse data. However, the difference between the source and target data distribution weakens the transferability. To bridge this gap, we focus on selecting source instances that are closely related to and have the same distribution as the target domain. In this paper, we propose a novel Adaptive Clustering Transfer Learning (ACTL) method to improve transferability. Specifically, we simultaneously train the instance selector and the transfer learning model. The selector adaptively conducts clustering on the training data and learns the weights for source instances. The weight will activate or inhibit the contribution of the corresponding source instance during transfer learning. Meanwhile, the transfer learning model guides the selector to learn the weight appropriately according to the objective function. To evaluate the effectiveness of our method, we conduct experiments on two different tasks including recommender system and text matching. Experimental results show that our method consistently outperforms competing methods and the selected source instances share a similar data distribution with the target domain.