面向约束语言模型策略优化的逐步对齐方法
Stepwise Alignment for Constrained Language Model Policy Optimization
AI总结：
本文提出SACPO算法，将人类价值对齐形式化为安全约束下的奖励最大化问题，通过逐步对齐奖励与安全指标，在理论上保证最优性和约束违反上界，并在Alpaca-7B上取得优于现有方法的有用性与无害性表现。
摘要：
安全性和可信赖性是使用大语言模型（LLMs）的AI系统在现实世界应用中不可或缺的要求。本文将人类价值对齐形式化为语言模型策略的优化问题，即在安全约束下最大化奖励，然后提出一种算法，即面向约束策略优化的逐步对齐（SACPO）。SACPO背后的一个关键思想（有理论支持）是，同时考虑奖励和安全的最优策略可以直接从奖励对齐的策略中获得。基于这一关键思想，SACPO逐步将LLMs与每个指标对齐，同时利用简单而强大的对齐算法，如直接偏好优化（DPO）。SACPO具有若干优势，包括简单性、稳定性、计算效率以及算法和数据集的灵活性。在温和假设下，我们的理论分析给出了最优性和安全约束违反的上界。我们的实验结果表明，SACPO在有用性和无害性方面都能比最先进的方法更好地微调Alpaca-7B。
英文摘要：
Safety and trustworthiness are indispensable requirements for real-world applications of AI systems using large language models (LLMs). This paper formulates human value alignment as an optimization problem of the language model policy to maximize reward under a safety constraint, and then proposes an algorithm, Stepwise Alignment for Constrained Policy Optimization (SACPO). One key idea behind SACPO, supported by theory, is that the optimal policy incorporating reward and safety can be directly obtained from a reward-aligned policy. Building on this key idea, SACPO aligns LLMs step-wise with each metric while leveraging simple yet powerful alignment algorithms such as direct preference optimization (DPO). SACPO offers several advantages, including simplicity, stability, computational efficiency, and flexibility of algorithms and datasets. Under mild assumptions, our theoretical analysis provides the upper bounds on optimality and safety constraint violation. Our experimental results show that SACPO can fine-tune Alpaca-7B better than the state-of-the-art method in terms of both helpfulness and harmlessness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
