Direct Preference Optimization (DPO) has recently emerged as a popular approach to improve reinforcement learning from human feedback (RLHF), leading to better techniques to fine-tune large language models (LLM). A weakness of DPO, however, lies in its lack of capability to characterize the diversity of human preferences. Inspired by Mallows' theory of preference ranking, we develop in this paper a new approach, the *MallowsPO*. A distinct feature of this approach is a *dispersion index*, which reflects the dispersion of human preference to prompts. We show that existing DPO models can be reduced to special cases of this dispersion index, thus unified with MallowsPO. More importantly, we demonstrate empirically how to use this dispersion index to enhance the performance of DPO in a broad array of benchmark tasks, from synthetic bandit selection to controllable generation and dialogues, while maintaining great generalization capabilities. MallowsPO is also compatible with other SOTA offline preference optimization methods, boosting nearly 2\% extra LC win rate when used as a plugin for fine-tuning Llama3-Instruct.

Recently, numerous preference optimization algorithms have been introduced as extensions to the Direct Preference Optimization (DPO) family. While these methods have successfully aligned models with human preferences, there is a lack of understanding regarding the contributions of their additional components. Moreover, fair and consistent comparisons are scarce, making it difficult to discern which components genuinely enhance downstream performance. In this work, we propose RainbowPO, a unified framework that demystifies the effectiveness of existing DPO methods by categorizing their key components into seven broad directions. We integrate these components into a single cohesive objective, enhancing the performance of each individual element. Through extensive experiments, we demonstrate that RainbowPO outperforms existing DPO variants. Additionally, we provide insights to guide researchers in developing new DPO methods and assist practitioners in their implementations.

Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area uses a discrete-time formulation, which is prone to induced errors, and often not applicable to models with higher-order/black-box solvers. The objective of this study is to develop a disciplined approach to fine-tuning diffusion models using continuous-time RL, formulated as a stochastic control problem with a reward function that aligns the end result (terminal state) with input prompt. The key idea is to treat score matching as controls or actions, and thereby connecting to policy optimization and regularization in continuous-time RL. To carry out this idea, we lay out a new policy optimization framework for continuous-time RL, and illustrate its potential in enhancing the value networks design space via leveraging the structural property of diffusion models. We validate the advantages of our method by experiments in downstream tasks of fine-tuning large-scale Text2Image models, Stable Diffusion v1.5.

Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area. Additionally, we provide a GitHub link https://github.com/hanyang1999/Preference-Tuning-with-Human-Feedback.

We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective), and show how it can be effectively used to derive performance difference and local approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.

Accurate prediction of a patient's length-of-stay (LOS) in the hospital enables an efficient and effective management of hospital beds. This paper studies LOS prediction for pediatric patients with respiratory diseases using three decision tree methods: Bagging, Adaboost, and Random forest. A data set of 11,206 records retrieved from the hospital information system is used for analysis after preprocessing and transformation through a computation and an expansion method. Two tests, namely bisection test and periodic test, are designed to assess the performance of the prediction methods. Bagging shows the best result on the bisection test (0.296 RMSE, 0.831 R2, and 0.723 Acc ± 1) for the testing set of the whole data test. The performances of the three methods are similar on the periodic test, whereas Adaboost performs slightly better than the other two methods. Results indicate that the three methods are all effective for the LOS prediction. This study also investigates the importance of different data fields to the LOS prediction, and finds that hospital treatment-related data fields contribute more to the LOS prediction than other categories of fields.

In a P2P system, a client peer may select one or more server peers to download a specific file. In a P2P resource economy, the server peers charge the client for the downloading. A server peer's price would naturally depend on the specific object being downloaded, the duration of the download, and the rate at which the download is to occur. The optimal peer selection problem is to select, from the set of peers that have the desired object, the subset of peers and download rates that minimizes cost. In this paper we examine a number of natural peer selection problems for both P2P downloading and P2P streaming. For downloading, we obtain the optimal solution for minimizing the download delay subject to a budget constraint, as well as the corresponding Nash equilibrium. For the streaming problem, we obtain a solution that minimizes cost subject to continuous playback while allowing for one or more server peers to fail during the streaming process. The methodologies developed in this paper are applicable to a variety of P2P resource economy problems.

We present a novel method for "what's next" scheduling of semiconductor manufacturing lines based on the deterministic fluid network model of Chen and Yao. By "what's next" scheduling we mean specifying the order in which jobs are processed at each tool group. The approach we take here is first to determine how to allocate tool capacity among competing job types by solving a series of linear and quadratic programming problems related to the fluid model and then to specify a "what's next" scheduling algorithm designed to track these capacity allocations. The primary advantage of our approach is that it gives rise to a dynamic schedule which is based on global rather than local state information and which is responsive to stochastic changes in the line including tool incapacitation events and operator unavailability. This is in contrast to many other scheduling rules, such as first come first served and closest to completion which are essentially static rules based on local information. We also present some theoretical results about the fluid model that have important implications in the context of semiconductor manufacturing as well as some preliminary simulation results comparing the fluid model to other scheduling policies.>

A broadcast communication network that interconnects multiple hosts N considered.A job generated by a host is either generic, m which case It can be processed at any of the hosts in the network, or dedicated, m which case it must be processed at the host from which it originates.It is assumed that each host can generate several types of dedicated jobs, where each type imposes a constraint on its average response time.The optimal design objective is twofold: First, to redistribute the generic jobs among the host computers in order to minimize the average response time, which shall be referred to as the load-balancing problem; second, for each host computer, to design a control that schedules the dedicated and the generic Jobs so that the response-time constraints are met for each of the dedicated traffic types.Despite its complexity, the model described above has some attractive features.Specifically, for a given allocation of the generic traffic, the scheduhng problem at each host can be solved as a polymatroid optimization problem.The underlying polymatroid structure leads to an efficient algorithm to determine, as a function of the offered load of generic traffic, the average delay of the generic jobs at any given host.Further analysis reveals that the delay functions at each of the hosts are convex and increasing.Therefore, an allocation algorlthm can be employed to solve the system-level load-balancing problem.An example is provided mdlcating that a substantial improvement m performance can be obtained by incorporating scheduling into the load-balancing procedure.

A stochastic system is considered in equilibrium with N servers, no waiting room, and K classes of customers. A class-k customer requires b/sub k/ servers and releases them simultaneously after a random period of time. This multiclass blocking system is motivated by loss networks that support a variety of traffic types (e.g. voice, video, facsimile). The effect of increasing the state-dependent arrival rates and the number of servers on the throughputs and blocking probabilities is considered. It is noted that the theory developed can be extended to the case where queueing is permitted in the knapsack.>

A flexible machine that processes a set of different part types is considered. Each part type has its own Poisson arrival process, general processing requirement, and a given delay (flow time) constraint. The problem is to find an optimal control that schedules the machine among the part types on a real-time basis in order to minimize a cost function of the flow times and meet all the constraints. The cost function is quite general, and it requires only that the marginal costs can be ordered among the part types. Based on polymatroid theory, an optimal control is identified that partitions the part types into several groups and applies a priority rule among the groups. Within each group, the optimal control is an appropriate randomization among priority rules. An efficient algorithm is developed to characterize the optimal control completely.>

The authors have developed and applied probabilistic tools to establish second-order stochastic properties such as convexity and concavity in the optimal design and control of queueing systems. Their approaches are based on constructing and comparing the dynamical sample paths of the stochastic processes under study. They present a tutorial on these notions and results in stochastic convexity/concavity. Many examples are discussed to illustrate the application of the results in parametric optimization of queues and queueing networks.>

Considered is a set of manufacturing cells that are jointly setup for production. Each cell is capable of producing a specific type of part family and periodically a joint setup is performed at each of these cells for production. We model the production process of each cell by a self-exciting point process. Using some structural properties of this process, an efficient algorithm is developed to obtain the optimal run quantity and the inter-setup time.

A new approach, i.e. the view through the fuzzy normal forms is presented that differs from previous approaches. The authors' concern is based on the fact that the implication as well as other connectives are linguistic entities that are inescapably associated with fuzziness. They are interested in the manner in which they introduce, or rather, add fuzziness into the statement in which they are included. New insight is brought into the structure of the `generalized modus ponens by their representation'.