Recently, ChatGPT or InstructGPT like large language models (LLM) has made a significant impact in the AI world. Many works have attempted to reproduce the complex InstructGPT's training pipeline, namely Reinforcement Learning with Human Feedback (RLHF). However, the mainstream distributed RLHF training methods typically adopt a fixed model placement strategy, referred to as the All-colocated strategy. This strategy treats all four interdependent models involved in RLHF as a single entity, distributing them across all devices and applying parallelism techniques designed for a single model, regardless of the workload heterogeneity inherent to each model. As a result, this strategy exacerbates memory constraints and generation bottlenecks in RLHF training. To address these issues, we propose a flexible model placement framework that offers two general and agile model placement strategies. The Interleaving strategy helps improve memory utilization and communication efficiency of RLHF training by placing models without dependencies on exclusive devices with careful orchestration. On the other hand, the Disaggregated strategy improves the throughput of model training by separating the training and inference runtime of the RLHF pipeline with additional shadow models. Furthermore, our framework provides a simple user interface and guidelines to easily and flexibly configure these strategies in various training scenarios. Our experiments have shown that our strategy can achieve notable improvements up to$12\times$, compared to Deepspeed-Chat or trlX. The results highlight the effectiveness and adaptability of our methods in accelerating the training of distributed RLHF.

The data sparsity problem has been a long-standing obstacle towards achieving better recommendation performance since it is miserable to estimate the user’s interests from limited historical behaviors. The pre-training paradigm, i.e., learning universal knowledge across a wide spectrum of domains, has increasingly become a new de-facto practice in many fields, especially for adaption to new domains. The merit of this superior generalizability renders it a natural choice to tackle the data sparsity problem for various recommendation scenarios. Hence, several efforts mainly follow masked language modeling or simple data augmentation via contrastive learning to build a pre-trained recommendation model. Our recent work (namely Miracle ) suggests that the common treatment utilizing the masked language modeling is not sufficient for pre-training a recommender system, since a user’s intent could be more complex than predicting the next word or item. The encouraging results demonstrate that the multi-interest modeling could significantly push the frontier of recommender system pre-training. Nevertheless, how to accommodate the temporal dynamics of the user interests seems to be underexplored under both single vector representation and multi-interest schemes. In this article, we aim to incorporate sophisticated temporal information modeling with the current advance in this line. More specifically, we extend Miracle by further considering relative position information and two kinds of relative time interval information jointly when performing multi-interest learning. Then, a sequential process for interest refinement is proposed to learn the subtle nuances of how interests change and shift along the timeline, leading to a more precise representation of user interests. Our extensive experiments on multiple real-world datasets validate the effectiveness of the proposed solution, demonstrating a significant improvement over current state-of-the-art models on these benchmarks. The code is available at https://github.com/WHUIR/Horae .

Cross-Domain Sequential Recommendation (CDSR) aims to enhance personalized user experiences by leveraging user behaviors across multiple domains. Existing methods primarily focus on fusing information from various domains and modeling global user preferences, but often struggle with negative transfer, where knowledge from one domain impairs recommendation performance in another. For example, a user may enjoy watching sports games in the video domain but have no interest in participating in sports activities. Consequently, this interest does not extend to purchasing related sports gear. In such cases, a recommendation system suggesting sports gear based on the user’s viewing preferences may not elicit a positive response. To tackle this issue, we propose a novel method called Multi-Interest Bridge Recommender (MIBR). In light of the cross-domain scenario, where user preferences are not entirely consistent across domains, we design a Multi-Interest Extraction (MIE) module to capture the diversity of user interests based on a soft clustering approach. In the meantime, we design a cross-domain bridging (CDB) module, with the goal of mitigating the issue of negative transfer. CDB leverages the extracted interests as a bridge for inter-domain information transfer, enabling each domain to adaptively extract relevant information from diverse interests while ignoring unrelated ones. Extensive experiments on three popular datasets reveal MIBR’s significant superiority over baselines, e.g., with up to a 59.27% uplift in terms of HR@10 over C2DSR on the Movie-Book dataset.

With the escalating complexity and investment cost of training deep neural networks, safeguarding them from unauthorized usage and intellectual property theft has become imperative. Especially the rampant misuse of prediction APIs to replicate models without access to the original data or architecture poses grave security threats. Diverse defense strategies have emerged to address these vulnerabilities, yet these defenses either incur heavy inference overheads or assume idealized attack scenarios. To address these challenges, we revisit the utilization of noise transition matrix as an efficient perturbation technique, which injects noise into predicted posteriors in a linear manner and integrates seamlessly into existing systems with minimal overhead, for model stealing defense. Provably, with such perturbed posteriors, the attacker's cloning process degrades into learning from noisy data. Toward optimizing the noise transition matrix, we proposed a novel bi-level optimization training framework, which performs fidelity on the victim model while the surrogate model adversarially. Comprehensive experimental results demonstrate that our method effectively thwarts model stealing attacks and achieves minimal utility tradeoffs, outperforming existing state-of-the-art defenses.

Sequential data mining is ubiquitous in various scenarios. Modeling event sequence and predicting event occurrence is of vital importance in sequential data mining, and Temporal Point Processes (TPP) are widely used in this area. Conventional TPP use objective functions as sum of classification loss for event type and regression loss for occurrence time, leading to practical limitations that conventional TPP is unable to predict the occurrence of each type of event and distinguish the dependency within and between different event types. To tackle these defects, we propose a Multi-task Decouple Learning (MTDL) framework to model TPP from a novel perspective of Multi-task Learning (MTL), i.e., predicting the next-step occurrence time for all event types using a weighted multi-task regression loss. We experiment with three state-of-the-arts, showing that the proposed MTDL framework can improve the performance of original TPP models. Moreover, we develop a Hierarchical Attentive Point Process (HAPP) to further exploit the potential of the proposed MTDL framework, using a hierarchical attention mechanism to capture the inner-sequence time dependency within the same type of events and the inter-sequence dependency between different types of events. Experiments on real-world business dataset and public datasets show the efficacy of the proposed method.

User behavior representation learned by self-supervised pre-training tasks is widely used in various domains and applications. Conventional methods usually follow the methodology in Natural Language Processing (NLP) to set the pre-training tasks. They either randomly mask some of the behaviors in the sequence and predict the masked ones or predict the next k behaviors. These methods fit for text sequence, in which the tokens are sequentially arranged subject to linguistic criterion. However, the user behavior sequences can be stochastic with noise and randomness. The same paradigm is intractable for learning a robust user behavioral representation.

We focus on the problem of event sequence clustering with different temporal patterns from the view of Reinforcement Learning (RL), whereby the observed sequences are assumed to be generated from a mixture of latent policies. We propose an Expectation-Maximization (EM) based algorithm to cluster the sequences with different temporal patterns into the underlying policies while simultaneously learning each of the policy model, in E-step estimating the cluster labels for each sequence, in M-step learning the respective policy. For each policy learning, we resort to Inverse Reinforcement Learning (IRL) by decomposing the observed sequence into states (hidden embedding of event history) and actions (time interval to next event) in order to learn a reward function. Experiments on synthetic and real-world datasets show the efficacy of our method against the state-of-the-arts.

Temporal point process is widely used for sequential data modeling. In this article, we focus on the problem of modeling sequential event propagation in graph, such as retweeting by social network users and news transmitting between websites. Given a collection of event propagation sequences, the conventional point process model considers only the event history, i.e., embed event history into a vector, not the latent graph structure. We propose a graph biased temporal point process (GBTPP) leveraging the structural information from graph representation learning, where the direct influence between nodes and indirect influence from event history is modeled. Moreover, the learned node embedding vector is also integrated into the embedded event history as side information. Experiments on a synthetic data set and two real-world data sets show the efficacy of our model compared with conventional methods and state-of-the-art ones.

Temporal point process (TPP) is an expressive tool for modeling the temporal pattern of event sequences. However, discovering temporal patterns for event sequences clustering is rarely studied in TPP modeling. To solve this problem, we take a reinforcement learning view whereby the observed sequences are assumed to be generated from a mixture of latent policies. The purpose is to cluster the sequences with different temporal patterns into the underlying policies while learning each of the policy model. The flexibility of our model lies in: i) all the components are networks including the policy network for modeling the temporal point process; ii) to handle varying-length event sequences, we resort to inverse reinforcement learning by decomposing the observed sequence into states (RNN hidden embedding of history) and actions (time interval to next event) in order to learn a reward function, it helps to achieve better performance or increasing efficiency compared to existing methods using rewards over the entire sequence such as log-likelihood or Wasserstein distance. We adopt an Expectation-Maximization algorithm, in E-step estimating the cluster labels for each sequence, in M-step aiming to learn the respective policy. Extensive experiments on synthetic and real-world datasets show the efficacy of our method against the state-of-the-arts.

Transfer learning leverages knowledge from a source domain with rich data to a target domain with sparse data. However, the difference between the source and target data distribution weakens the transferability. To bridge this gap, we focus on selecting source instances that are closely related to and have the same distribution as the target domain. In this paper, we propose a novel Adaptive Clustering Transfer Learning (ACTL) method to improve transferability. Specifically, we simultaneously train the instance selector and the transfer learning model. The selector adaptively conducts clustering on the training data and learns the weights for source instances. The weight will activate or inhibit the contribution of the corresponding source instance during transfer learning. Meanwhile, the transfer learning model guides the selector to learn the weight appropriately according to the objective function. To evaluate the effectiveness of our method, we conduct experiments on two different tasks including recommender system and text matching. Experimental results show that our method consistently outperforms competing methods and the selected source instances share a similar data distribution with the target domain.

This paper presents a factorial marked temporal point process model and presents efficient learning methods. In conventional (multi-dimensional) marked temporal point process models, an event is often encoded by a single discrete variable (marker). We describe the factorial marked point processes whereby time-stamped event is factored into multiple markers. Accordingly the size of the infectivity matrix modeling the effect between pairwise markers is in exponential order regarding the number of discrete markers. We propose a decoupled learning method with two learning procedures: i) directly solving the model based on two techniques: Alternating Direction Method of Multipliers and Fast Iterative Shrinkage-Thresholding Algorithm; ii) involving a reformulation that transforms the original problem into a Logistic Regression model for more efficient learning. Moreover, a sparse group regularizer is added to identify the key profile features and event labels. Empirical results on real world datasets demonstrate the efficiency of our decoupled and reformulated method.

We focus on an important problem of predicting the so-called “patient flow” from longitudinal electronic health records (EHRs), which has not been explored via existing machine learning techniques. We develop a point process based framework for modeling patient flow through various care units (CUs) and jointly predicting patients' destination CUs and duration days. We propose a novel discriminative learning algorithm aiming at improving the prediction of transition events in the case of sparse data. By parameterizing the proposed model as mutually-correcting processes, we formulate the estimation problem via generalized linear models and solve it based on alternating direction method of multipliers (ADMM). We achieve simultaneous feature selection and learning by adding a group-lasso regularizer to the ADMM algorithm. Additionally, we synthesize auxiliary training data for the classes with extremely few samples, and improve the robustness of our learning method to the problem of data imbalance.

Over the past decade, the rate of care unit (CU) use in the United States has been increasing. With an aging population and ever-growing demand for medical care, effective management of patients' transitions among different care facilities will prove indispensible for shortening the length of hospital stays, improving patient outcomes, allocating critical care resources, and reducing preventable re-admissions. In this paper, we focus on an important problem of predicting the so-called “patient flow” from longitudinal electronic health records (EHRs), which has not been explored via existing machine learning techniques. By treating a sequence of transition events as a point process, we develop a novel framework for modeling patient flow through various CUs and jointly predicting patients' destination CUs and duration days. Instead of learning a generative point process model via maximum likelihood estimation, we propose a novel discriminative learning algorithm aiming at improving the prediction of transition events in the case of sparse data. By parameterizing the proposed model as a mutually-correcting process, we formulate the estimation problem via generalized linear models, which lends itself to efficient learning based on alternating direction method of multipliers (ADMM). Furthermore, we achieve simultaneous feature selection and learning by adding a group-lasso regularizer to the ADMM algorithm. Additionally, for suppressing the negative influence of data imbalance on the learning of model, we synthesize auxiliary training data for the classes with extremely few samples, and improve the robustness of our learning method accordingly. Testing on real-world data, we show that our method obtains superior performance in terms of accuracy of predicting the destination CU transition and duration of each CU occupancy.