Reinforcement Learning from Human Feedback (RLHF)has been crucial to the recent success of Large Language Models (LLMs), however it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimized the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where the performance as measured by the learned proxy reward model increases, but the true model quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs), such as Direct Preference Optimization (DPO) have emerged as alternatives to the classical RLHF pipeline. However, despite not training a separate proxy reward model or using RL, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL-budgets, DAA algorithms exhibit similar degradation patters to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL-budgets, but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation this work formulates the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.

Over the last decades, repeat-pass SAR interferometry (InSAR) for deformation measurement and topographic mapping has revolutionized our understanding of many geophysical processes on Earth. A new mission concept, currently in development at the Jet Propulsion Laboratory (JPL) and Caltech, aims at using orbital repeat-pass InSAR for deformation and topography mapping of Saturn’s ice-covered and geologically active moon Enceladus. In this paper, we present an initial performance assessment of the system and the suggested SAR processing approach, along with simulated InSAR acquisitions using a DLR in-house End-to-End performance simulator.