IterAlign：大型语言模型的迭代宪法对齐
IterAlign: Iterative Constitutional Alignment of Large Language Models
AI总结：
提出IterAlign框架，通过红队测试和强模型自动发现宪法并迭代指导基础LLM自我纠正，无需大量人工标注，在多个安全基准上显著提升模型对齐效果。
摘要：
随着大型语言模型（LLMs）的快速发展，将LLMs与人类价值观和社会规范对齐以确保其可靠性和安全性变得至关重要。已提出基于人类反馈的强化学习（RLHF）和Constitutional AI（CAI）用于LLM对齐。然而，这些方法要么需要大量人类标注，要么需要显式预定义的宪法，这既耗费人力又消耗资源。为克服这些缺点，我们研究了基于宪法的LLM对齐，并提出了一种数据驱动的宪法发现与自对齐框架，称为IterAlign。IterAlign利用红队测试揭示LLM的弱点，并使用更强的LLM自动发现新宪法。这些宪法随后被用于指导基础LLM的自我纠正。这种宪法发现流程可以迭代且自动地运行，以发现专门针对当前LLM中对齐差距的新宪法。在多个安全基准数据集和多个基础LLM上的实证结果表明，IterAlign成功提升了真实性、有用性、无害性和诚实性，在无害性方面将LLM对齐提升了高达13.5%。
英文摘要：
With the rapid development of large language models (LLMs), aligning LLMs with human values and societal norms to ensure their reliability and safety has become crucial. Reinforcement learning with human feedback (RLHF) and Constitutional AI (CAI) have been proposed for LLM alignment. However, these methods require either heavy human annotations or explicitly pre-defined constitutions, which are labor-intensive and resource-consuming. To overcome these drawbacks, we study constitution-based LLM alignment and propose a data-driven constitution discovery and self-alignment framework called IterAlign. IterAlign leverages red teaming to unveil the weaknesses of an LLM and automatically discovers new constitutions using a stronger LLM. These constitutions are then used to guide self-correction of the base LLM. Such a constitution discovery pipeline can be run iteratively and automatically to discover new constitutions that specifically target the alignment gaps in the current LLM. Empirical results on several safety benchmark datasets and multiple base LLMs show that IterAlign successfully improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to $13.5\%$ in harmlessness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
