Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed *Self-Play Preference Optimization* (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. Additionally, we propose a new SPPO objective which is both strongly motivated by theory and is simple and effective in practice. In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53\% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench, Arena-Hard, and the Open LLM Leaderboard. Starting from a stronger base model Llama-3-8B-Instruct, we are able to achieve a length-controlled win rate of 38.77\%. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models.

Recent studies have shown that the regret of reinforcement learning (RL) can be polylogarithmic in the planning horizon $H$. However, it remains an open question whether such a result holds for adversarial RL. In this paper, we answer this question affirmatively by proposing the first horizon-free policy search algorithm. To tackle the challenges caused by exploration and adversarially chosen reward over episodes, our algorithm employs (1) a variance-uncertainty-aware weighted least square estimator for the transition kernel; and (2) an occupancy measure-based technique for the online search of a stochastic policy. We show that our algorithm achieves an $\tilde{O}\big((d+\log |\mathcal{S}|)\sqrt{K} + d^2\big)$ regret with full-information feedback, where $d$ is the dimension of a known feature mapping linearly parametrizing the unknown transition kernel of the MDP, $K$ is the number of episodes, $|\mathcal{S}|$ is the cardinality of the state space. We also provide hardness results to justify the near optimality of our algorithm and the inevitability of $\log|\mathcal{S}|$ in the regret bound.

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM’s performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents.

Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images (``winner'' and ``loser'' images) for each text prompt. In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process. Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms the existing supervised fine-tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHF-based methods across all metrics, achieving these results with less data. Codes are available at \url{https://github.com/uclaml/SPIN-Diffusion/}.

The occurrence of extreme events, which has increased the risk of major outages in the grid, makes the quick and efficient recovery of load in the distribution network become a key issue. The data-driven deep reinforcement learning method has great potential in providing fast decision-making. However, a large number of agents lead to the curse of dimensionality, making it inefficient to obtain effective control strategies. When tackling similar tasks in different power gird, the retraining of multiple agents will bring us great cost. To solve this problem, we propose a transferable multi-agent reinforcement learning framework that employs model reload and buffer reuse methods to transfer control strategies from small-scale simple scenes to large-scale complex scenes. It also utilizes attention mechanisms to aggregate observation features and handles the problem of variable observation dimensions. Finally, the distribution service restoration problem is modeled as a Markov decision process and solved using the QMIX algorithm. The performance of the proposed method has been verified in IEEE 34-node and IEEE 123-node distribution systems.

Cloud data centers provide end-users with a wide range of application scenarios, including scientific computing, smart grids, etc. The number and size of data centers have rapidly increased in recent years, which causes severe environmental problems and colossal power demand. Therefore, it is desirable to use a proper scheduling method to optimize resource usage and reduce energy consumption in a data center. However, it is rather difficult to design an effective and efficient task scheduling algorithm because of the dynamic and complex environment of data centers. This paper proposes a task scheduling algorithm, WSS, to optimize resource usage and reduce energy consumption based on a model-free deep reinforcement learning framework inspired by the Wolpertinger architecture. The proposed algorithm can handle the scheduling problem on a sizeable discrete action space, improve decision efficiency, and save the training convergence time. Meanwhile, the proposed algorithm based on Soft Actor-Critic is designed to improve the stability and exploration capability of WSS. Experiments based on real-world traces prove that WSS can reduce energy consumption by nearly 25% compared with the Deep Q-network task scheduling algorithm. Moreover, WSS can provide a short time of training convergence without increasing the average waiting time of tasks and achieve stable performance.

The size and amount of colocation data centers (colocations, for short) have been growing rapidly with the increasing popularity of cloud services, which ultimately causes a heavy burden on the power grid and the environment. However, even a colocation owner wishes to reduce its energy consumption, it may not be able to apply some effective energy saving techniques directly to the servers, since the servers belong to and are operated by its tenants. To solve the "uncoordinated relationship" issue between owners and tenants and achieve the energy reduction with a limited cost budget, an energy saving-oriented incentive mechanism, ESCo is proposed in this paper. Different from existing mechanisms that emphasize on minimizing the cost for the owners, our mechanism emphasizes on the maximization of the amount of energy saved by the tenants given a limited budget that the owner wants to pay for the energy saving. An algorithm is developed to realize a stable assignment in the mechanism. Trace-driven simulations based on real-world data are performed to verify the effectiveness of ESCo. The results show that ESCo can achieve 14.47% more energy saving than existing colocation incentive mechanisms.

Colocation data centers (colocations, for short) are developing rapidly in recent years, resulting in a heavy burden on the power grid and the environment. Due to the special management mode of colocations, even the colocation operators wish to reduce their power demand, they have no authority to control the servers because the servers belong to and are operated by the tenants themselves. To solve the "uncoordinated relationship" issue between operators and tenants, a truthful and feasible incentive mechanism MesPP is proposed in this paper. Different from existing works, MesPP aims at maximizing the energy reduction of colocations with a limited cost budget and can be applied even there is no demand response (DR) program. Meanwhile, power prediction is integrated into MesPP to further improve the energy efficiency of colocations and fairness of the mechanism. To solve the optimization problem, we develop a (1−ε)-approximation algorithm. Simulations are performed and show that MesPP can achieve 12.23% more energy saving compared with existing incentive mechanisms.

Vehicular ad hoc network (VANET) is a promising technology for intelligent transportation systems, where dedicated short range communication (DSRC) is usually used for inter-vehicle communications. When the vehicle density is high, the randomly access feature of the carrier sense multiple access with collision avoidance (CSMA/CA) mechanism in DSRC will lead to the increased channel contention delay. Therefore, visible light communication (VLC) can be introduced to form a heterogeneous VLC-DSRC network for delay reduction. Although VLC has no contention delay, one VLC link can only be established between two adjacent vehicles, which might increase the delay caused by multi-hop communications. In this paper, a delay-minimization link selection scheme is proposed to select appropriate links for vehicles according to actual situations and minimize average delay. Simulation results demonstrate that the proposed scheme outperforms the considered benchmarks in terms of average transmission delay.

In indoor environment, a hybrid network consisting of light fidelity (LiFi) and wireless fidelity (WiFi) is capable of retaining both the high-speed data transmission and the ubiquitous coverage, where the network access type of users can be optimized to improve the performance. Since visible light communication mainly depends on the line-of-sight (LoS) transmission, it is susceptible to channel blockage, which should be considered by users to select the appropriate type of network access. In the existing literature, LiFi channel blockage parameters are regarded as known for users to determine the access type. However, in practical scenarios, the estimation of blockage parameters lags behind their variations, and users can not get the real-time blockage information. In this paper, a support-vector-machine- based (SVM-based) network access type decision scheme is proposed in hybrid LiFi and WiFi networks. By taking the correlation of blockage parameters between adjacent periods into account, SVM is adopted to achieve high equivalent data rate when accurate blockage parameters are unknown. Simulation results demonstrate that the proposed scheme outperforms the considered benchmarks under different scenarios in terms of the equivalent data rate performance.