对抗评论家自博弈：面向语言模型的可证明且可扩展的离线对齐方法
Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
AI总结：
本文提出SPAC方法，将自博弈与平均悲观技术结合，首次实现可证明且可扩展的LLM离线偏好对齐，并在7B Mistral模型上验证了其有效性。
摘要：
本文研究将大型语言模型（LLMs）与离线偏好数据对齐的挑战，特别聚焦于基于人类反馈的强化学习（RLHF）对齐方法。尽管当前流行的偏好优化方法在实践中表现出良好的经验性能，但根据经典离线强化学习（RL）的结果，它们在理论上并不能保证收敛到最优策略，并且在数据覆盖稀疏时可证明会失败。另一方面，近期一系列工作专注于具有理论动机且具备可证明保证的偏好优化方法，但这些方法对于LLM对齐等大规模应用而言计算效率不足。为弥合这一差距，我们提出SPAC，一种受离线RL文献中平均悲观技术启发的、采用自博弈的新型离线偏好优化方法，旨在成为首个可证明且可扩展的LLM对齐方法。我们既在一般函数逼近设定下、单策略集中性条件下为其收敛性提供了理论分析，又通过在7B Mistral模型上结合Open LLM Leaderboard评估，展示了其在LLM对齐任务中具有竞争力的经验性能。
英文摘要：
This work studies the challenge of aligning large language models (LLMs) with offline preference data. We focus on alignment by Reinforcement Learning from Human Feedback (RLHF) in particular. While popular preference optimization methods exhibit good empirical performance in practice, they are not theoretically guaranteed to converge to the optimal policy and can provably fail when the data coverage is sparse by classical offline reinforcement learning (RL) results. On the other hand, a recent line of work has focused on theoretically motivated preference optimization methods with provable guarantees, but these are not computationally efficient for large-scale applications like LLM alignment. To bridge this gap, we propose SPAC, a new offline preference optimization method with self-play, inspired by the on-average pessimism technique from the offline RL literature, to be the first provable and scalable approach to LLM alignment. We both provide theoretical analysis for its convergence under single-policy concentrability for the general function approximation setting and demonstrate its competitive empirical performance for LLM alignment on a 7B Mistral model with Open LLM Leaderboard evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
