This is a corrigendum for the article “One Model for All: Large Language Models are Domain-Agnostic Recommendation Systems” published in ACM Trans. Inf. Syst . 43, 5, Article 118 (July 2025), 27 pages.

The data sparsity problem has been a long-standing obstacle towards achieving better recommendation performance since it is miserable to estimate the user’s interests from limited historical behaviors. The pre-training paradigm, i.e., learning universal knowledge across a wide spectrum of domains, has increasingly become a new de-facto practice in many fields, especially for adaption to new domains. The merit of this superior generalizability renders it a natural choice to tackle the data sparsity problem for various recommendation scenarios. Hence, several efforts mainly follow masked language modeling or simple data augmentation via contrastive learning to build a pre-trained recommendation model. Our recent work (namely Miracle ) suggests that the common treatment utilizing the masked language modeling is not sufficient for pre-training a recommender system, since a user’s intent could be more complex than predicting the next word or item. The encouraging results demonstrate that the multi-interest modeling could significantly push the frontier of recommender system pre-training. Nevertheless, how to accommodate the temporal dynamics of the user interests seems to be underexplored under both single vector representation and multi-interest schemes. In this article, we aim to incorporate sophisticated temporal information modeling with the current advance in this line. More specifically, we extend Miracle by further considering relative position information and two kinds of relative time interval information jointly when performing multi-interest learning. Then, a sequential process for interest refinement is proposed to learn the subtle nuances of how interests change and shift along the timeline, leading to a more precise representation of user interests. Our extensive experiments on multiple real-world datasets validate the effectiveness of the proposed solution, demonstrating a significant improvement over current state-of-the-art models on these benchmarks. The code is available at https://github.com/WHUIR/Horae .

Sequential recommendation systems aim to predict users’ next likely interaction based on their history. However, these systems face data sparsity and cold-start problems. Utilizing data from other domains, known as multi-domain methods, is useful for alleviating these problems. However, traditional multi-domain methods rely on meaningless ID-based item representation, which makes it difficult to align items with similar meanings from different domains, yielding sup-optimal knowledge transfer. This article introduces LLM-Rec , a framework that utilizes pre-trained Large Language Models (LLMs) for domain-agnostic recommendation. Specifically, we mix user’s behaviors from multiple domains and concatenate item titles into a sentence, then use LLMs for generating user and item representations. By mixing behaviors across different domains, we can exploit the knowledge encoded in LLMs to bridge the semantic across over multi-domain behaviors, thus obtaining semantically rich representations and improving performance in all domains. Furthermore, we explore the underlying reasons why LLMs are effective and investigate whether LLMs can understand the semantic correlations as the recommendation model, and if advanced techniques like scaling laws in NLP also work in recommendations. We conduct extensive experiments with LLMs ranging from 40 M to 6.7 B to answer the above questions and to verify the effectiveness of LLM-Rec in multi-domain recommendation. The source code is available at https://github.com/WHUIR/LLMRec .

Recently, the item textual information has been exploited with pre-trained language models (PLMs) to enrich the representations of tail items. The underlying idea is to align the hot items and tail items in terms of the external semantic knowledge covered by the PLM. However, it is non-trivial to eliminate the popularity bias by exploiting the textual semantics. One major obstacle is that the model supervision still counts on the sparse yet binary user behaviors. In the preliminary investigation, we discover that text-based recommendations also suffer from the popularity bias.

Many distributed training techniques like Parameter Server and AllReduce have been proposed to take advantage of the increasingly large data and rich features. However, stragglers frequently occur in distributed training due to resource contention and hardware heterogeneity, which significantly hampers the training efficiency. Previous works only address part of the stragglers and could not adaptively solve various stragglers in practice. Additionally, it is challenging to use a systematic framework to address all stragglers because different stragglers require diverse data allocation and fault-tolerance mechanisms. Therefore, this paper proposes a unified distributed training framework called AntDT (Ant Distributed Training Framework) to adaptively solve the straggler problems. Firstly, the framework consists of four components, including the Stateful Dynamic Data Sharding service, Monitor, Controller, and Agent. These components work collaboratively to efficiently distribute workloads and provide a range of pre-defined straggler mitigation methods with fault tolerance, thereby hiding messy details of data allocation and fault handling. Secondly, the framework provides a high degree of flexibility, allowing for the customization of straggler mitigation solutions based on the specific circumstances of the cluster. Leveraging this flexibility, we introduce two straggler mitigation solutions, namely AntDT-ND for non-dedicated clusters and AntDT-DD for dedicated clusters, as practical examples to resolve various types of stragglers at Ant Group. Justified by our comprehensive experiments and industrial deployment statistics, AntDT outperforms other SOTA methods more than 3 × in terms of training efficiency. Additionally, in Alipay's homepage recommendation scenario, using AntDT reduces the training duration of the ranking model from 27.8 hours to just 5.4 hours.

With the rise of large language models (LLMs), recent works have leveraged LLMs to improve the performance of click-through rate (CTR) prediction. However, we argue that a critical obstacle remains in deploying LLMs for practical use: the efficiency of LLMs when processing long textual user behaviors. As user sequences grow longer, the current efficiency of LLMs is inadequate for training on billions of users and items. To break through the efficiency barrier of LLMs, we propose Behavior Aggregated Hierarchical Encoding (BAHE) to enhance the efficiency of LLM-based CTR modeling. Specifically, BAHE proposes a novel hierarchical architecture that decouples the encoding of user behaviors from inter-behavior interactions. Firstly, to prevent computational redundancy from repeated encoding of identical user behaviors, BAHE employs the LLM's pre-trained shallow layers to extract embeddings of the most granular, atomic user behaviors from extensive user sequences and stores them in the offline database. Subsequently, the deeper, trainable layers of the LLM facilitate intricate inter-behavior interactions, thereby generating comprehensive user embeddings. This separation allows the learning of high-level user representations to be independent of low-level behavior encoding, significantly reducing computational complexity. Finally, these refined user embeddings, in conjunction with correspondingly processed item embeddings, are incorporated into the CTR model to compute the CTR scores. Extensive experimental results show that BAHE reduces training time and memory by five times for CTR models using LLMs, especially with longer user sequences. BAHE has been deployed in a real-world system, allowing for daily updates of 50 million CTR data on 8 A100 GPUs, making LLMs practical for industrial CTR prediction.

Click-through rate (CTR) prediction plays a crucial role in recommendation systems, with significant impact on user experience and platform revenue generation. Despite the various public CTR datasets available due to increasing interest from both academia and industry, these datasets have limitations. They cover a limited range of scenarios and predominantly focus on ID-based features, neglecting the vital role of multi-modal features for effective multi-scenario CTR prediction. Moreover, their scale is modest compared to real-world industrial datasets, hindering robust and comprehensive evaluation of complex models. To address these challenges, we introduce a large-scale Multi-Scenario Multi-Modal CTR dataset named AntM2 C, built from real industrial data from Alipay. This dataset offers an impressive breadth and depth of information, covering CTR data from four diverse business scenarios, including advertisements, consumer coupons, mini-programs, and videos. Unlike existing datasets, AntM2 C provides not only ID-based features but also five textual features and one image feature for both users and items, supporting more delicate multi-modal CTR prediction. AntM2 C is also substantially larger than existing datasets, comprising 100 million CTR data. This scale allows for robust and comprehensive evaluation and comparison of CTR prediction models. We employ AntM2 C to construct several typical CTR tasks, including multi-scenario modeling, item and user cold-start modeling, and multi-modal modeling. Initial experiments and comparisons with baseline methods have shown that AntM2 C presents both new challenges and opportunities for CTR models, with the potential to significantly advance CTR research. The AntM2 C dataset is available at https://www.atecup.cn/OfficalDataSet.

Recently, a new paradigm, meta learning, has been widely applied to Deep Learning Recommendation Models (DLRM) and significantly improves statistical performance, especially in cold-start scenarios. However, the existing systems are not tailored for meta learning based DLRM models and have critical problems regarding efficiency in distributed training in the GPU cluster. It is because the conventional deep learning pipeline is not optimized for two task-specific datasets and two update loops in meta learning. This paper provides a high-performance framework for large-scale training for Optimization-based Meta DLRM models over the G PU cluster, namely G -Meta. Firstly, G-Meta utilizes both data parallelism and model parallelism with careful orchestration regarding computation and communication efficiency, to enable high-speed distributed training. Secondly, it proposes a Meta-IO pipeline for efficient data ingestion to alleviate the I/O bottleneck. Various experimental results show that G-Meta achieves notable training speed without loss of statistical performance. Since early 2022, G-Meta has been deployed in Alipay's core advertising and recommender system, shrinking the continuous delivery of models by four times. It also obtains 6.48% improvement in Conversion Rate (CVR) and 1.06% increase in CPM (Cost Per Mille) in Alipay's homepage display advertising, with the benefit of larger training samples and tasks.

Industrial recommender systems usually need to serve multiple scenarios at the same time. In practice, there are various heterogeneous scenarios, since users frequently engage in scenarios with varying intentions and the items within each scenario typically belong to diverse categories. Existing works of multi-scenario recommendation mainly focus on modeling homogeneous scenarios which have similar data distributions. They equally transfer knowledge to each scenario without considering the diversity of heterogeneous scenarios. In this paper, we argue that the heterogeneity in multi-scenario recommendations is a key problem that needs to be solved. To this end, we propose an industrial framework named Scenario-Aware Model-Agnostic Meta Distillation (SAMD) for the multi-scenario recommendation. SAMD aims to provide scenario-aware and model-agnostic knowledge sharing across heterogeneous scenarios by modeling scenarios' relationship and conducting heterogeneous knowledge distillation. Specifically, SAMD first measures the comprehensive representation of each scenario and then proposes a novel meta distillation paradigm to conduct scenario-aware knowledge sharing. The meta network first establishes the potential scenarios' relationships and generates the strategies of knowledge sharing for each scenario. Then the heterogeneous knowledge distillation utilizes scenario-aware strategies to share knowledge across heterogeneous scenarios through intermediate features distillation without the restriction of the model architecture. In this way, SAMD shares knowledge across heterogeneous scenarios in a scenario-aware and model-agnostic manner, which addresses the problem of heterogeneity. Compared with other state-of-the-art methods, extensive offline experiments, and online A/B testing demonstrate the superior performance of the proposed SAMD framework, especially in heterogeneous scenarios.

The cold-start problem of conversion rate prediction is a common challenge in online advertising systems. To alleviate this problem, a large number of methods either use content information or uncertainty methods, or use meta-learning based methods to improve the ranking performance of cold-start items. However, they can work for cold-start scenarios but fail to adaptively unify warm and cold recommendations into one model, requiring additional human efforts or knowledge to adapt to different scenarios. Meanwhile, none of them pay attention to the discrepancy between model predictions and true likelihoods of cold items, while over- or under-estimation is harmful to the ROI (Return on Investment) of advertising placements. In this paper, in order to address the above issues, we propose a framework called Distribution-Constrained Batch Transformer (DCBT). Specifically, the framework introduces a Transformer module into the batch dimension to automatically choose proper information from warm samples to enhance the representation of cold samples and preserve the property of warm samples. In addition, to avoid the distribution of cold samples being affected by the warm samples, the framework adds MMD loss to constrain the sample distribution before and after feeding into the Transformer module. Extensive offline experiments on two real-world datasets show that our proposed method attains state-of-the-art performance in AUC and PCOC (Predicted CVR over CVR) for cold items and warm items. An online A/B test demonstrates that the DCBT model obtained a 20.08% improvement in CVR and a 13.21% increase in GMV (Gross Merchandise Volume).

In recommender systems, content-based methods and meta-learning involved methods usually have been adopted to alleviate the item cold-start problem. The former consider utilizing item attributes at the feature level and the latter aim at learning a globally shared initialization for all tasks to achieve fast adaptation with limited data at the task level. However, content-based methods only focus on the similarity of item attributes, ignoring the relationships established by user interactions. And for tasks with different distributions, most meta-learning-based methods are difficult to achieve better performance under a single initialization. To address the limitations mentioned above and combine the strengths of both methods, we propose a Task Similarity Aware Meta-Learning (TSAML) framework from two aspects. Specifically, at the feature level, we simultaneously introduce content information and user-item relationships to exploit task similarity. At the task level, we design an automatic soft clustering module to cluster similar tasks and generate the same initialization for similar tasks. Extensive offline experiments demonstrate that the TSAML framework has superior performance and recommends cold items to preferred users more effectively than other state-of-the-art methods.

There exists the cold-start problem in the recommendation systems when observed user-item interactions are insufficient. To alleviate this problem, most existing works aim to learn globally shared prior knowledge across all items and be fast adapted to a new item with few interactions. However, such learning techniques are data demanding and work poorly on new items with no interactions. In this applied paper, we present an industrial framework recently deployed on Alipay to address the item cold-start problem in zero-shot scenarios. The proposed framework provides both efficient and high-quality recommendations for cold items with no log data. Specifically, we formulate the cold-start problem as a zero-shot learning problem and build a highly efficient infrastructure to accomplish online zero-shot recommendations used on large-scale platforms. Extensive offline experiments and online A/B testing demonstrate that the proposed framework has superior performance and recommends cold items to preferred users more effectively than other state-of-the-art methods.

Multi-task learning(MTL) is an open and challenging problem in various real-world applications. The typical way of conducting multi-task learning is establishing some global parameter sharing mechanism across all tasks or assigning each task an individual set of parameters with cross-connections between tasks. However, for most existing approaches, all tasks just thoroughly or proportionally share all the features without distinguishing the helpfulness of them. By that, some tasks would be intervened by the unhelpful features that are useful for other tasks, leading to undesired negative transfer between tasks. In this paper, we design a novel architecture named the Multiple-level Sparse Sharing Model (MSSM), which can learn features selectively and share knowledge across all tasks efficiently. MSSM first employs a field-level sparse connection module (FSCM) to enable much more expressive combinations of feature fields to be learned for generalization across tasks while still allowing for task-specific features to be customized for each task. Furthermore, a cell-level sparse sharing module (CSSM) can recognize the sharing pattern through a set of coding variables that selectively choose which cells to route for a given task. Extensive experimental results on several real-world datasets show that MSSM outperforms SOTA models significantly in terms of AUC and LogLoss metrics.

Transfer learning leverages knowledge from a source domain with rich data to a target domain with sparse data. However, the difference between the source and target data distribution weakens the transferability. To bridge this gap, we focus on selecting source instances that are closely related to and have the same distribution as the target domain. In this paper, we propose a novel Adaptive Clustering Transfer Learning (ACTL) method to improve transferability. Specifically, we simultaneously train the instance selector and the transfer learning model. The selector adaptively conducts clustering on the training data and learns the weights for source instances. The weight will activate or inhibit the contribution of the corresponding source instance during transfer learning. Meanwhile, the transfer learning model guides the selector to learn the weight appropriately according to the objective function. To evaluate the effectiveness of our method, we conduct experiments on two different tasks including recommender system and text matching. Experimental results show that our method consistently outperforms competing methods and the selected source instances share a similar data distribution with the target domain.