Pre-ranking plays a crucial role in large-scale recommender systems by significantly improving the efficiency and scalability within the constraints of providing high-quality candidate sets in real time. The two-tower model is widely used in pre-ranking systems due to a good balance between efficiency and effectiveness with decoupled architecture, which independently processes user and item inputs before calculating their interaction (e.g. dot product or similarity measure). However, this independence also leads to the lack of information interaction between the two towers, resulting in less effectiveness. In this paper, a novel architecture named learnable Fully Interacted Two-tower Model (FIT) is proposed, which enables rich information interactions while ensuring inference efficiency. FIT mainly consists of two parts: Meta Query Module (MQM) and Lightweight Similarity Scorer (LSS). Specifically, MQM introduces a learnable item meta matrix to achieve expressive early interaction between user and item features. Moreover, LSS is designed to further obtain effective late interaction between the user and item towers. Finally, experimental results on several public datasets show that our proposed FIT significantly outperforms the state-of-the-art baseline pre-ranking models.

Recommender systems are widely used in various online platforms. In the context of sequential recommendation, it is essential to accurately capture the chronological patterns in user activities to generate relevant recommendations. Conventional ID-based sequential recommenders have shown promise but lack comprehensive real-world knowledge about items, limiting their effectiveness. Recent advancements in Large Language Models (LLMs) offer the potential to bridge this gap by leveraging the extensive real-world knowledge encapsulated in LLMs. However, integrating LLMs into sequential recommender systems comes with its own challenges, including inadequate representation of sequential behavior patterns and long inference latency. In this paper, we propose SeRALM (Enhancing Sequential Recommenders with Augmented Knowledge from Aligned Large Language Models) to address these challenges. SeRALM integrates LLMs with conventional ID-based sequential recommenders for sequential recommendation tasks. We combine text-format knowledge generated by LLMs with item IDs and feed this enriched data into ID-based recommenders, benefitting from the strengths of both paradigms. Moreover, we develop a theoretically underpinned alignment training method to refine LLMs' generation using feedback from ID-based recommenders for better knowledge augmentation. We also present an asynchronous technique to expedite the alignment training process. Experimental results on public benchmarks demonstrate that SeRALM significantly improves the performances of ID-based sequential recommenders. Further, a series of ablation studies and analyses corroborate SeRALM's proficiency in steering LLMs to generate more pertinent and advantageous knowledge across diverse scenarios.

In this work, we share our experience on tele-knowledge pre-training for fault analysis, a crucial task in telecommunication applications that requires a wide range of knowledge normally found in both machine log data and product documents. To organize this knowledge from experts uniformly, we propose to create a Tele-KG (tele-knowledge graph). Using this valuable data, we further propose a tele-domain language pre-training model TeleBERT and its knowledge-enhanced version, a tele-knowledge re-training model KTeleBERT. which includes effective prompt hints, adaptive numerical data encoding, and two knowledge injection paradigms. Concretely, our proposal includes two stages: first, pre-training TeleBERT on 20 million tele-related corpora, and then re-training it on 1 million causal and machine-related corpora to obtain KTeleBERT. Our evaluation on multiple tasks related to fault analysis in tele-applications, including root-cause analysis, event association prediction, and fault chain tracing, shows that pretraining a language model with tele-domain data is beneficial for downstream tasks. Moreover, the KTeleBERT re-training further improves the performance of task models, highlighting the effectiveness of incorporating diverse tele-knowledge into the model.

Graph convolutional networks (GCNs) play a critical role in improving the performance of collaborative filtering. They leverage the concept of aggregating neighbor information to capture user preferences on bipartite graphs by stacking multiple convolutional layers. However, this requirement for layer stacking often leads to a long training time for convergence, and results in indistinguishable representations with significant performance deterioration due to the problem of oversmoothing. Additionally, the noise of interactions will be amplified by the stacking of convolutional layers through message passing. To address these issues, we propose a simple, plug-and-play-Neighborhood Structure -Embedding approach, named NSE, which utilizes first-order adjacency information to construct structural embeddings. By explicitly incorporating local topologically statistical information before message passing, the embeddings propagated at GCNs have better topology-structure awareness. This leads to an improved optimization path and greater robustness against noise propagation. Experimental results demonstrate significant performance improvements by employing our proposed NSE in graph collaborative filtering models. Particularly, the NSE-enhanced LGCN shows performance gains of 5.06% and 4.86% on the Yelp and Amazon-Books datasets, respectively. The average training convergence speed is improved by 204.8%. NSE-enhanced graph collaborative filtering has also demonstrated excellent robustness against both noise and oversmoothing.

Prediction models for click-through rate (CTR) learn feature interactions underlying user behaviors, which are crucial in recommendation systems. Due to their size and complexity, existing approaches have a limited range of applications. In order to decrease inference delay, knowledge distillation techniques have been used in recommendation systems. Due to the student model's lower capacity, the knowledge distillation process is less effective when there is a significant difference in the complexity of the network architecture between the teacher model and the student model.

A session-based recommendation system (SRS) tries to predict the next possible choice of anonymous users. In recent years, graph neural network (GNN) models have been successfully applied to SRSs and have achieved great success. Using GNN models in SRSs, each session graph is processed successively to obtain the embedding of the node (i.e, each action on an item), which is then imported into the prediction module to generate recommendation results. However, solely depending on the session graph to obtain the node embeddings is not sufficient because each session only involves a few items. Therefore, neighbor sessions have been used to extend the session graph to learn more informative node representations. In this paper, we introduce a S ession-based recommendation MO del based on N eighbor sessions with similar probabilistic int E ntions(SMONE). SMONE models the intentions behind sessions in a probabilistic way and retrieves the neighbor sessions with similar intentions. After the neighbor sessions are found, the target session and its neighbor sessions are modeled as a hypyergraph to learn the contextualized embeddings, which are combined with item embeddings through GNN to produce the final item recommendations. Experiments on real-world datasets prove the effectiveness and superiority of SMONE.

Multi-task learning (MTL) has been successfully applied to a wide range of real-world applications. However, MTL models often suffer from performance degradation with negative transfer due to sharing all features without distinguishing their helpfulness for all tasks. To this end, many works on feature selection for multi-task learning (FS-MTL) have been proposed to alleviate negative transfer between tasks by learning features selectively for each specific task. However, due to latent confounders between features and task targets, the correlations captured by the feature selection modules proposed in these works may fail to reflect the actual effect of the features on the targets. This paper explains negative transfer in FS-MTL from a causal perspective and presents a novel architecture called Causal Feature Selection for Multi-task Learning(CFS-MTL). This method incorporates the idea of causal inference into feature selection for multi-task learning via pseudo-intervention. It aims to select features with more stable causal effects rather than spurious correlations for each task by regularizing the distance between feature ITEs and feature importance. We conduct extensive experiments based on three real-world datasets to demonstrate that our proposed CFS-MTL outperforms state-of-the-art MTL models significantly in the AUC metric.

Multi-task learning (MTL) aims at solving multiple related tasks simultaneously and has experienced rapid growth in recent years. However, MTL models often suffer from performance degeneration with negative transfer due to learning several tasks simultaneously. Some related work attributed the source of the problem is the conflicting gradients. In this case, it is needed to select useful gradient updates for all tasks carefully. To this end, we propose a novel optimization approach for MTL, named GDOD, which manipulates gradients of each task using an orthogonal basis decomposed from the span of all task gradients. GDOD decomposes gradients into task-shared and task-conflict components explicitly and adopts a general update rule for avoiding interference across all task gradients. This allows guiding the update directions depending on the task-shared components. Moreover, we prove the convergence of GDOD theoretically under both convex and non-convex assumptions. Experiment results on several multi-task datasets not only demonstrate the significant improvement of GDOD performed to existing MTL models but also prove that our algorithm outperforms state-of-the-art optimization methods in terms of AUC and Logloss metrics.

Multi-task learning (MTL) has experienced rapid growth in recent years. A typical way of conducting MTL with deep neural networks (DNNs) is either establishing a sort of global feature sharing mechanism across all tasks or assigning each task an individual set of parameters with cross-connections. However, these existing approaches leverage DNNs only to share features of a certain order. Several modelsdemonstrated that explicitly modeling feature sharing with both low-order and high-order features can boost performance. To this end, we propose a model-agnostic sparse routing architecture called MASR, which emphasizes arbitrary order feature sharing for multi-task learning. It is able to choose specific orders of features to route for a given task through learnable latent variables. Moreover, MASR is model-agnostic and can be combined with existing MTL models to share features of both low-order and high-order. Extensive experimental results on several real-world datasets not only confirm the significant improvement of MASR performed to existing MTL models but also outperform existing hybrid architectures in terms of AUC metric.

Click-through rate (CTR) prediction is a long-standing problem in advertising systems. Existing single-domain CTR prediction methods suffer from the data sparsity problem since few users can click advertisements on many items. Recently, cross-domain CTR prediction leverages the relatively richer information from a source domain to improve the performance on a target domain with sparser information, but it cannot explicitly capture users' diverse interests in different domains. In this paper, we propose a novel hierarchical user behavior modeling framework for cross-domain CTR prediction, named HBMNet. HBMNet contains two main components: an element-wise behavior transfer(EWBT) layer and a user representation layer. EWBT layer transfers the information collected from one domain by element-level masks to dynamically highlight the informative elements in another domain. The user representation layer performs behavior-level attention between these behavior representations and the ranking item representation. Extensive experimental results on two cross-domain datasets show that the proposed HBMNet outperforms SOTA models.

Recent methods in sequential recommendation focus on learning multi-interest embedding vectors from a user's behavior sequence for the next-item recommendation. However, behavior sequential data may result from users' conformity towards popular items, which entangles users' real interests and tends to recommend popular items by using interest embeddings. In this paper, we propose a novel multi-interest framework with causal embedding for sequential recommendation, called MiceRec. Specifically, we first obtain two embedding layers from behavior sequence by assigning items with separate embeddings for interest and conformity, then extract multiple pure interests from one embedding layer, while the other for users' conformity extraction. According to the colliding effect of causal inference, we mine cause-specific data for training causal embeddings. Our framework significantly outperforms state-of-the-art solutions on two real-world datasets1. We further demonstrate that the learned multi-interest embeddings successfully separate from each other, and show that conformity information is almost squeezed out from interest embeddings.

Cross-domain recommendation can help alleviate the data sparsity issue in traditional sequential recommender systems. In this paper, we propose the RecGURU algorithm framework to generate a Generalized User Representation (GUR) incorporating user information across domains in sequential recommendation, even when there is minimum or no common users in the two domains. We propose a self-attentive autoencoder to derive latent user representations, and a domain discriminator, which aims to predict the origin domain of a generated latent representation. We propose a novel adversarial learning method to train the two modules to unify user embeddings generated from different domains into a single global GUR for each user. The learned GUR captures the overall preferences and characteristics of a user and thus can be used to augment the behavior data and improve recommendations in any single domain in which the user is involved. Extensive experiments have been conducted on two public cross-domain recommendation datasets as well as a large dataset collected from real-world applications. The results demonstrate that RecGURU boosts performance and outperforms various state-of-the-art sequential recommendation and cross-domain recommendation methods. The collected data will be released to facilitate future research.

Multi-task learning(MTL) is an open and challenging problem in various real-world applications. The typical way of conducting multi-task learning is establishing some global parameter sharing mechanism across all tasks or assigning each task an individual set of parameters with cross-connections between tasks. However, for most existing approaches, all tasks just thoroughly or proportionally share all the features without distinguishing the helpfulness of them. By that, some tasks would be intervened by the unhelpful features that are useful for other tasks, leading to undesired negative transfer between tasks. In this paper, we design a novel architecture named the Multiple-level Sparse Sharing Model (MSSM), which can learn features selectively and share knowledge across all tasks efficiently. MSSM first employs a field-level sparse connection module (FSCM) to enable much more expressive combinations of feature fields to be learned for generalization across tasks while still allowing for task-specific features to be customized for each task. Furthermore, a cell-level sparse sharing module (CSSM) can recognize the sharing pattern through a set of coding variables that selectively choose which cells to route for a given task. Extensive experimental results on several real-world datasets show that MSSM outperforms SOTA models significantly in terms of AUC and LogLoss metrics.

Deep learning models for human activity recognition (HAR) based on sensor data have been heavily studied recently. However, the generalization ability of deep models on complex real-world HAR data is limited by the availability of high-quality labeled activity data, which are hard to obtain. In this article, we design a similarity embedding neural network that maps input sensor signals onto real vectors through carefully designed convolutional and Long Short-Term Memory (LSTM) layers. The embedding network is trained with a pairwise similarity loss, encouraging the clustering of samples from the same class in the embedded real space, and can be effectively trained on a small dataset and even on a noisy dataset with mislabeled samples. Based on the learned embeddings, we further propose both nonparametric and parametric approaches for activity recognition. Extensive evaluation based on two public datasets has shown that the proposed similarity embedding network significantly outperforms state-of-the-art deep models on HAR classification tasks, is robust to mislabeled samples in the training set, and can also be used to effectively denoise a noisy dataset.