Modern LLMs face a major obstacle: each new pre-trained model version requires expensive and repetitive alignment.We propose a method that transfers fine-tuning updates across model versions.The key idea is to extract the diff vector, which is the difference in parameters induced by fine-tuning, from a source model version and apply it to the base of a different target version.We show that transferring diff vectors significantly improves the target base model, often achieving performance comparable to its fine-tuned counterpart.For example, applying the fine-tuning updates from Llama 3.0 8B to Llama 3.1 8B increases accuracy by 46.9% on IFEval and 15.7% on Live-CodeBench without further training, surpassing Llama 3.1 8B Instruct.In multilingual settings, we also observe accuracy gains relative to Llama 3.1 8B Instruct, including 4.7% for Malagasy and 15.5% for Turkish on Global MMLU.Our controlled experiments reveal that fine-tuning transfer works best when source and target models are linearly connected in parameter space.We also show that this transfer provides a stronger and more efficient starting point for subsequent fine-tuning.Finally, we propose an iterative recycling-then-finetuning approach for continuous model development, which improves both efficiency and effectiveness.Our findings suggest that fine-tuning transfer is a viable strategy to reduce training costs while maintaining model performance.1

We study reward poisoning attacks on Combinatorial Multi-armed Bandits (CMAB). We first provide a sufficient and necessary condition for the attackability of CMAB, a notion to capture the vulnerability and robustness of CMAB. The attackability condition depends on the intrinsic properties of the corresponding CMAB instance such as the reward distributions of super arms and outcome distributions of base arms. Additionally, we devise an attack algorithm for attackable CMAB instances. Contrary to prior understanding of multi-armed bandits, our work reveals a surprising fact that the attackability of a specific CMAB instance also depends on whether the bandit instance is known or unknown to the adversary. This finding indicates that adversarial attacks on CMAB are difficult in practice and a general attack strategy for any CMAB instance does not exist since the environment is mostly unknown to the adversary. We validate our theoretical findings via extensive experiments on real-world CMAB applications including probabilistic maximum covering problem, online minimum spanning tree, cascading bandits for online ranking, and online shortest path.

Off-policy Learning to Rank (LTR) aims to optimize a ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how users generate the click data, i.e., the click model, and hence need to tailor their methods specifically under different click models. In this paper, we unified the ranking process under general stochastic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which could be easily applied to a wide range of click models. Through a dedicated formulation of the MDP, we show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms while maintaining consistency and robustness under different click models.

We introduce a risk-aware variant of the Traveling Salesperson Problem (TSP), where the robot tour cost and reward have to be optimized simultaneously, while being subjected to uncertainty in both. We study the case where the rewards and the costs exhibit diminishing marginal gains, i.e., are submodular. Since the costs and the rewards are stochastic, we seek to maximize a risk metric known as Conditional-Value-at-Risk (CVaR) of the submodular function. We propose a Risk-Aware Greedy Algorithm (RAGA) to find an approximate solution for this problem. The approximation algorithm runs in polynomial time and is within a constant factor of the optimal and an additive term that depends on the value of optimal solution. We use the submodular function’s curvature to improve approximation results further and verify the algorithm’s performance through simulations.