QA-LIGN：通过宪法分解的问答对对齐大语言模型
QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
AI总结：
QA-LIGN通过分解奖励信号提升LLM对齐效果，降低攻击成功率并保持低拒绝率，实现安全与帮助性的帕累托最优。
摘要：
大型语言模型（LLMs）与帮助性、诚实性和无害性等原则的对齐通常依赖于标量奖励，这些奖励会掩盖驱动训练信号的目标。我们引入了QA-LIGN，通过结构化的自然语言程序将单一奖励分解为可解释的原则特定评估。模型通过草稿、批评和修订的流程进行学习，其中符号评估对照标准为初始和修订响应提供透明的反馈，在GRPO训练中。应用于未受限制的Llama-3.1-8B-Instruct，QA-LIGN将攻击成功率降低至最高68.7%，同时保持0.67%的虚假拒绝率，实现了帕累托最优的安全-帮助性性能，并在同等训练条件下优于DPO和GRPO。这些结果表明，使奖励信号可解释和模块化可以提高对齐效果，表明透明性增强了LLM的安全性。
英文摘要：
Alignment of large language models (LLMs) with principles like helpfulness, honesty, and harmlessness typically relies on scalar rewards that obscure which objectives drive the training signal. We introduce QA-LIGN, which decomposes monolithic rewards into interpretable principle-specific evaluations through structured natural language programs. Models learn through a draft, critique, and revise pipeline, where symbolic evaluation against the rubrics provides transparent feedback for both initial and revised responses during GRPO training. Applied to uncensored Llama-3.1-8B-Instruct, QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate, achieving Pareto optimal safety-helpfulness performance and outperforming both DPO and GRPO with state-of-the-art reward models given equivalent training. These results demonstrate that making reward signals interpretable and modular improves alignment effectiveness, suggesting transparency enhances LLM safety.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
