Existing preference optimization objectives for language model alignment require additional hyperparameters that must be extensively tuned to achieve optimal performance, increasing both the complexity and time required for fine-tuning large language models. In this paper, we propose a simple yet effective hyperparameter-free preference optimization algorithm for alignment. We observe that promising performance can be achieved simply by optimizing inverse perplexity, which is calculated as the inverse of the exponentiated average log-likelihood of the chosen and rejected responses in the preference dataset. The resulting simple learning objective, SimPER, is easy to implement and eliminates the need for expensive hyperparameter tuning and a reference model, making it both computationally and memory efficient. Extensive experiments on widely used real-world benchmarks, including MT-Bench, AlpacaEval 2, and 10 key benchmarks of the Open LLM Leaderboard with 5 base models, demonstrate that SimPER consistently and significantly outperforms existing approaches—even without any hyperparameters or a reference model. For example, despite its simplicity, SimPER outperforms state-of-the-art methods by up to 5.7 points on AlpacaEval 2 and achieves the highest average ranking across 10 benchmarks on the Open LLM Leaderboard. The source code for SimPER is publicly available at: https://github.com/tengxiao1/SimPER.

This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks.

This paper introduces a novel generalized selfimitation learning (GSIL) framework, which effectively and efficiently aligns large language models with offline demonstration data.We develop GSIL by deriving a surrogate objective of imitation learning with density ratio estimates, facilitating the use of self-generated data and optimizing the imitation learning objective with simple classification losses.GSIL eliminates the need for complex adversarial training in standard imitation learning, achieving lightweight and efficient fine-tuning for large language models.In addition, GSIL encompasses a family of offline losses parameterized by a general class of convex functions for density ratio estimation and enables a unified view for alignment with demonstration data.Extensive experiments show that GSIL consistently and significantly outperforms baselines in many challenging benchmarks, such as coding (HuamnEval), mathematical reasoning (GSM8K) and instruction-following benchmark (MT-Bench).Code is public available at https://github.com/tengxiao1/GSIL.

We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.

Online travel platforms (OTPs), for example, booking.com, Ctrip.com, and Fliggy, deliver travel experiences to online users by providing travel-related products. Hotel recommendation is significantly important for OTPs since hotel bookings account for almost half of the travel expenses and hotel products generate more than half of OTP’s revenues. More than 58% Fliggy users may choose to use query-less hotel searches to find candidate hotels, where no additional keywords are given except the expected check-in date and travel destination city. Thus, how to recommend hotels to traveler users is important and challenging. In this article, we explore the unique characteristics of query-less hotel users and propose a novel multiscenario query-less search network (MSQS). According to their searching date, expected check-in date, current city, and expected hotel city, MSQS groups users’ behaviors (e.g., click, purchase, search) into four scenario groups, namely today-local, today-nonlocal, future-local, and future-nonlocal. The key components of MSQS are the global expert, the scenario expert, and the feedback expert. The global expert learns common features among different scenarios and extracts the feature interactions between context, users, and hotels. The scenario expert utilizes multilayer perception to learn the differentiating features between scenarios. The feedback expert learns users’ preferences for hotels in different scenarios through their historical behaviors, and a scenario interest extractor is carefully designed to enhance attention across scenarios and behaviors. An offline experiment on the Fliggy production dataset with over 8 million users and 0.49 million travel items and an online A/B test both show that MSQS effectively predicts users’ hotel booking intentions.