In recent years, there has been a trend in the field of recommender systems towards multi-task modeling and multi-scenario modeling. The aim is to enhance the performance of various tasks and scenarios by jointly training on multiple tasks or scenarios to learn common patterns and features. Joint modeling of tasks and scenarios has also received widespread attention recently. However, despite the rich proposals of methods for Multi-Task Learning (MTL), Multi-Scenario Learning (MSL), and Multi-Task-Multi-Scenario Learning (MTMSL) in recent years, there still lacks a comprehensive benchmark to evaluate these methods. Previous studies often employed different datasets, data processing techniques, data partitioning strategies, and hyperparameter settings, making replication of existing research and fair comparison of experimental results challenging. To address this challenge, we introduce MMLRec, the first unified comprehensive benchmark for evaluating MTL, MSL and MTMSL, featuring consistent dataset processing and identical parameter settings. This benchmark implements a range of MTL, MSL, and MTMSL algorithms, and evaluates them on multiple commonly used recommender systems datasets. Through fair comparative experiments, we find that some structurally simplistic recommendation algorithms are underestimated, as they can achieve comparable results to more complex algorithms while maintaining lower complexity. Furthermore, our experimental analysis indicates that more complex methods exhibit better robustness when there are significant differences between tasks or scenarios. By providing a unified framework (MMLRec), our goal is to promote rapid evaluation and inspire innovative research in this continuously evolving field. We hope that our open-source benchmark can facilitate swift, equitable evaluations, while also fostering further breakthrough research in the domains of MTL, MSL, and MTMSL.

Click-through rate (CTR) prediction plays a crucial role in recommendation systems, with significant impact on user experience and platform revenue generation. Despite the various public CTR datasets available due to increasing interest from both academia and industry, these datasets have limitations. They cover a limited range of scenarios and predominantly focus on ID-based features, neglecting the vital role of multi-modal features for effective multi-scenario CTR prediction. Moreover, their scale is modest compared to real-world industrial datasets, hindering robust and comprehensive evaluation of complex models. To address these challenges, we introduce a large-scale Multi-Scenario Multi-Modal CTR dataset named AntM2 C, built from real industrial data from Alipay. This dataset offers an impressive breadth and depth of information, covering CTR data from four diverse business scenarios, including advertisements, consumer coupons, mini-programs, and videos. Unlike existing datasets, AntM2 C provides not only ID-based features but also five textual features and one image feature for both users and items, supporting more delicate multi-modal CTR prediction. AntM2 C is also substantially larger than existing datasets, comprising 100 million CTR data. This scale allows for robust and comprehensive evaluation and comparison of CTR prediction models. We employ AntM2 C to construct several typical CTR tasks, including multi-scenario modeling, item and user cold-start modeling, and multi-modal modeling. Initial experiments and comparisons with baseline methods have shown that AntM2 C presents both new challenges and opportunities for CTR models, with the potential to significantly advance CTR research. The AntM2 C dataset is available at https://www.atecup.cn/OfficalDataSet.

Conversion rate (CVR) prediction models play a vital role in recommendation systems. Recent research shows that learning a unified model to serve multiple scenarios is effective for improving overall performance. However, it remains challenging to improve model prediction performance across scenarios at low model parameter cost, and current solutions are hard to robustly model multi-scenario diversity. In this paper, we propose MI-DPG for the multi-scenario CVR prediction, which learns scenario-conditioned dynamic model parameters for each scenario in a more efficient and effective manner. Specifically, we introduce an auxiliary network to generate scenario-conditioned dynamic weighting matrices, which are obtained by combining decomposed scenario-specific and scenario-shared low-rank matrices with parameter efficiency. For each scenario, weighting the backbone model parameters by the weighting matrix helps to specialize the model parameters for different scenarios. It can not only modulate the complete parameter space of the backbone model but also improve the model effectiveness. Furthermore, we design a mutual information regularization to enhance the diversity of model parameters across scenarios by maximizing the mutual information between the scenario-aware input and the scenario-conditioned dynamic weighting matrix. Experiments from three real-world datasets show that MI-DPG outperforms previous multi-scenario recommendation models.

In industrial recommendation systems, both data sizes and computational resources vary across different scenarios. For scenarios with limited data, data sparsity can lead to a decrease in model performance. Heterogeneous knowledge distillation-based transfer learning can be used to transfer knowledge from models in data-rich domains. However, in recommendation systems, the target domain possesses specific privileged features that significantly contribute to the model. While existing knowledge distillation methods have not taken these features into consideration, leading to suboptimal transfer weights. To overcome this limitation, we propose a novel algorithm called Uncertainty-based Heterogeneous Privileged Knowledge Distillation (UHPKD). Our method aims to quantify the knowledge of both the source and target domains, which represents the uncertainty of the models. This approach allows us to derive transfer weights based on the knowledge gain, which captures the difference in knowledge between the source and target domains. Experiments conducted on both public and industrial datasets demonstrate the superiority of our UHPKD algorithm compared to other state-of-the-art methods.

The cold-start problem of conversion rate prediction is a common challenge in online advertising systems. To alleviate this problem, a large number of methods either use content information or uncertainty methods, or use meta-learning based methods to improve the ranking performance of cold-start items. However, they can work for cold-start scenarios but fail to adaptively unify warm and cold recommendations into one model, requiring additional human efforts or knowledge to adapt to different scenarios. Meanwhile, none of them pay attention to the discrepancy between model predictions and true likelihoods of cold items, while over- or under-estimation is harmful to the ROI (Return on Investment) of advertising placements. In this paper, in order to address the above issues, we propose a framework called Distribution-Constrained Batch Transformer (DCBT). Specifically, the framework introduces a Transformer module into the batch dimension to automatically choose proper information from warm samples to enhance the representation of cold samples and preserve the property of warm samples. In addition, to avoid the distribution of cold samples being affected by the warm samples, the framework adds MMD loss to constrain the sample distribution before and after feeding into the Transformer module. Extensive offline experiments on two real-world datasets show that our proposed method attains state-of-the-art performance in AUC and PCOC (Predicted CVR over CVR) for cold items and warm items. An online A/B test demonstrates that the DCBT model obtained a 20.08% improvement in CVR and a 13.21% increase in GMV (Gross Merchandise Volume).

Conversion Rate (CVR) prediction is a crucial task in online advertising systems. Existing single-domain CVR prediction models suffer from the data sparsity problem since few users purchase items after clicking. In recent years, a robust and effective technique called fine-tuning can transfer knowledge from a data-rich source domain to enhance the CVR prediction performance in a data-sparse target domain. However, since most CVR prediction models have a large number of parameters, fine-tuning all the parameters on a data-sparse domain may lead to over-fitting. In this paper, we propose a general and efficient transfer learning method called Gradient-Flow based Fine-Tuning (GFlow-FT), which only needs to update a subset of parameters (called child network) via pruning the gradients to restrain gradient norm against over-fitting. In addition, our method employs the gradient-flow based measure via calculating the Hessian-gradient product as the criteria for picking the child network, which is superior to the magnitude-based and loss-based measure from empirical results. Extensive experimental results on three real-world datasets from recommendation systems show that GFlow-FT can significantly improve the performance of CVR prediction compared with state-of-the-art fine-tuning approaches.

In recommender systems, content-based methods and meta-learning involved methods usually have been adopted to alleviate the item cold-start problem. The former consider utilizing item attributes at the feature level and the latter aim at learning a globally shared initialization for all tasks to achieve fast adaptation with limited data at the task level. However, content-based methods only focus on the similarity of item attributes, ignoring the relationships established by user interactions. And for tasks with different distributions, most meta-learning-based methods are difficult to achieve better performance under a single initialization. To address the limitations mentioned above and combine the strengths of both methods, we propose a Task Similarity Aware Meta-Learning (TSAML) framework from two aspects. Specifically, at the feature level, we simultaneously introduce content information and user-item relationships to exploit task similarity. At the task level, we design an automatic soft clustering module to cluster similar tasks and generate the same initialization for similar tasks. Extensive offline experiments demonstrate that the TSAML framework has superior performance and recommends cold items to preferred users more effectively than other state-of-the-art methods.

Multi-task learning(MTL) is an open and challenging problem in various real-world applications. The typical way of conducting multi-task learning is establishing some global parameter sharing mechanism across all tasks or assigning each task an individual set of parameters with cross-connections between tasks. However, for most existing approaches, all tasks just thoroughly or proportionally share all the features without distinguishing the helpfulness of them. By that, some tasks would be intervened by the unhelpful features that are useful for other tasks, leading to undesired negative transfer between tasks. In this paper, we design a novel architecture named the Multiple-level Sparse Sharing Model (MSSM), which can learn features selectively and share knowledge across all tasks efficiently. MSSM first employs a field-level sparse connection module (FSCM) to enable much more expressive combinations of feature fields to be learned for generalization across tasks while still allowing for task-specific features to be customized for each task. Furthermore, a cell-level sparse sharing module (CSSM) can recognize the sharing pattern through a set of coding variables that selectively choose which cells to route for a given task. Extensive experimental results on several real-world datasets show that MSSM outperforms SOTA models significantly in terms of AUC and LogLoss metrics.

Transfer learning leverages knowledge from a source domain with rich data to a target domain with sparse data. However, the difference between the source and target data distribution weakens the transferability. To bridge this gap, we focus on selecting source instances that are closely related to and have the same distribution as the target domain. In this paper, we propose a novel Adaptive Clustering Transfer Learning (ACTL) method to improve transferability. Specifically, we simultaneously train the instance selector and the transfer learning model. The selector adaptively conducts clustering on the training data and learns the weights for source instances. The weight will activate or inhibit the contribution of the corresponding source instance during transfer learning. Meanwhile, the transfer learning model guides the selector to learn the weight appropriately according to the objective function. To evaluate the effectiveness of our method, we conduct experiments on two different tasks including recommender system and text matching. Experimental results show that our method consistently outperforms competing methods and the selected source instances share a similar data distribution with the target domain.