用于语言模型对齐与个性化的价值增强采样
Value Augmented Sampling for Language Model Alignment and Personalization
AI总结：
本文提出 VAS 框架，仅用冻结 LLM 的采样数据即可稳定优化奖励，无需访问或修改模型权重，性能优于 PPO、DPO，并支持 API 模型适配与部署时多奖励可控组合。
摘要：
使大型语言模型（LLM）能够迎合不同的人类偏好、学习新技能并忘却有害行为，是一个重要问题。基于搜索的方法（如 Best-of-N 或 Monte-Carlo Tree Search）性能出色，但由于推理成本高昂，并不适用于 LLM 适配。另一方面，使用强化学习（RL）进行适配在计算上高效，却因协同训练价值函数与策略时的优化难题而表现较差。我们提出了一个用于奖励优化的新框架 Value Augmented Sampling（VAS，价值增强采样），它仅利用从初始的、冻结的 LLM 采样的数据，即可最大化不同的奖励函数。VAS 求解最优的奖励最大化策略，而无需协同训练策略和价值函数，从而使优化保持稳定；它在标准基准测试上优于 PPO 和 DPO 等成熟基线，并以更低的推理成本取得了与 Best-of-128 相当的结果。与需要更改 LLM 权重的现有 RL 方法不同，VAS 不需要访问预训练 LLM 的权重。因此，它甚至可以适配仅以 API 形式提供的 LLM（例如 ChatGPT）。此外，我们的算法解锁了在部署时组合多种奖励并控制每种奖励程度的新能力，为对齐、个性化 LLM 的未来铺平了道路。
英文摘要：
Aligning Large Language Models (LLMs) to cater to different human preferences, learning new skills, and unlearning harmful behavior is an important problem. Search-based methods, such as Best-of-N or Monte-Carlo Tree Search, are performant, but impractical for LLM adaptation due to their high inference cost. On the other hand, using Reinforcement Learning (RL) for adaptation is computationally efficient, but performs worse due to the optimization challenges in co-training the value function and the policy. We present a new framework for reward optimization, Value Augmented Sampling (VAS), that can maximize different reward functions using data sampled from only the initial, frozen LLM. VAS solves for the optimal reward-maximizing policy without co-training the policy and the value function, making the optimization stable, outperforming established baselines, such as PPO and DPO, on standard benchmarks, and achieving comparable results to Best-of-128 with lower inference cost. Unlike existing RL methods that require changing the weights of the LLM, VAS does not require access to the weights of the pre-trained LLM. Thus, it can even adapt LLMs (e.g., ChatGPT), which are available only as APIs. In addition, our algorithm unlocks the new capability of composing several rewards and controlling the extent of each one during deployment time, paving the road ahead for the future of aligned, personalized LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
