基于一条通用原则的自参考 AI 反馈对齐大语言模型
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
AI总结：
提出自参考 AI 反馈框架，让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据，显著提升强化学习对齐效果。
摘要：
在对齐大语言模型（LLM）时，利用现有先进 AI 而非人类提供的反馈，是扩展监督信号的一种重要方法。然而，AI 很难理解人类意图和社会价值观，并据此提供准确的偏好反馈。当前的 AI 反馈方法依赖强大的 LLM 以及精心设计、用于描述人类意图的具体原则，而且容易受到位置偏差影响。为解决这些问题，我们提出一种基于自参考的 AI 反馈框架，使 13B Llama2-Chat 能够在“最有利于人类”等简单且通用的原则下提供高质量反馈。具体而言，我们让 AI 首先回应用户指令，然后以其自身回答为参考，生成对其他答案的批评，最后根据这些批评判断哪个答案更符合人类偏好。此外，我们使用自一致性方法进一步降低位置偏差的影响，并采用语义困惑度计算不同答案之间的偏好强度差异。实验结果表明，我们的方法使 13B 和 70B Llama2-Chat 标注器能够提供高质量偏好反馈，并且基于这些偏好数据通过强化学习训练出的策略模型在基准数据集上取得显著优势。
英文摘要：
In aligning large language models (LLMs), utilizing feedback from existing advanced AI rather than humans is an important method to scale supervisory signals. However, it is highly challenging for AI to understand human intentions and societal values, and provide accurate preference feedback based on these. Current AI feedback methods rely on powerful LLMs, carefully designed specific principles to describe human intentions, and are easily influenced by position bias. To address these issues, we propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback under simple and general principles such as ``best for humanity``. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference, and finally determine which answer better fits human preferences according to the criticism. Additionally, we use a self-consistency method to further reduce the impact of position bias, and employ semantic perplexity to calculate the preference strength differences between different answers. Experimental results show that our method enables 13B and 70B Llama2-Chat annotators to provide high-quality preference feedback, and the policy models trained based on these preference data achieve significant advantages in benchmark datasets through reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
