可视化策略-奖励相互作用以指导大型语言模型的零阶偏好优化
Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models
AI总结：
本文提出ZOPrO，通过分析策略与奖励模型更新关系并改进SPSA采样，首次将零阶优化用于LLM生成任务的偏好优化。
摘要：
使用反向传播等一阶方法微调大型语言模型（LLM）具有很高的计算强度。零阶（ZO）优化利用函数评估而非梯度，从而降低内存占用，但在高维模型中存在收敛缓慢的问题。因此，LLM中的ZO研究大多集中于分类任务，忽视了更复杂的生成式任务。本文提出ZOPrO，这是一种专为LLM中的偏好优化设计的新型ZO算法。我们首先分析传统（一阶）偏好优化过程中策略模型与奖励模型之间的相互作用，揭示二者相对更新中的模式。在这些洞见指导下，我们采用带定向采样策略的同时扰动随机逼近（SPSA）来加速收敛。通过在摘要生成、机器翻译和对话助手上的实验，我们证明该方法能够持续增强奖励信号，同时达到与一阶方法相当的收敛时间。尽管其尚不及部分最先进方法，但本工作首次将零阶方法应用于LLM中的偏好优化，超越了分类任务，并为一个在很大程度上尚未探索的研究方向铺平道路。代码和可视化可在https://github.com/alessioGalatolo/VisZOPrO获取。
英文摘要：
Fine-tuning Large Language Models (LLMs) with first-order methods like back-propagation is computationally intensive. Zeroth-Order (ZO) optimisation uses function evaluations instead of gradients, reducing memory usage, but suffers from slow convergence in high-dimensional models. As a result, ZO research in LLMs has mostly focused on classification, overlooking more complex generative tasks. In this paper, we introduce ZOPrO, a novel ZO algorithm designed for Preference Optimisation in LLMs. We begin by analysing the interplay between policy and reward models during traditional (first-order) Preference Optimisation, uncovering patterns in their relative updates. Guided by these insights, we adapt Simultaneous Perturbation Stochastic Approximation (SPSA) with a targeted sampling strategy to accelerate convergence. Through experiments on summarisation, machine translation, and conversational assistants, we demonstrate that our method consistently enhances reward signals while achieving convergence times comparable to first-order methods. While it falls short of some state-of-the-art methods, our work is the first to apply Zeroth-Order methods to Preference Optimisation in LLMs, going beyond classification tasks and paving the way for a largely unexplored research direction. Code and visualisations are available at https://github.com/alessioGalatolo/VisZOPrO
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
