重新思考对齐方法的评估：关于多样性、泛化与安全性的洞察
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
AI总结：
提出统一评估框架，跨事实性、安全性、简洁性、主动性和多样性五个维度，使用分布内外数据集比较PPO、DPO、ORPO、KTO等LLM对齐方法，揭示各方法权衡，指导更平衡可靠LLM开发。
摘要：
大型语言模型（LLMs）需要仔细的对齐以平衡相互竞争的目标——事实性、安全性、简洁性、主动性和多样性。现有研究侧重于单一技术或特定维度，缺乏对这些内在权衡的整体评估。我们提出了一个统一的评估框架，使用分布内和分布外数据集，跨这五个维度比较LLM对齐方法（PPO、DPO、ORPO、KTO）。利用通过人类研究验证的专用LLM-as-Judge提示词，我们发现DPO和KTO在事实准确性上表现突出，PPO和DPO在安全性上领先，而PPO在平衡简洁性与主动性方面表现最佳。我们的研究结果为常见对齐方法的权衡提供了见解，指导开发更平衡、更可靠的LLMs。
英文摘要：
Large language models (LLMs) require careful alignment to balance competing objectives - factuality, safety, conciseness, proactivity, and diversity. Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. We propose a unified evaluation framework that compares LLM alignment methods (PPO, DPO, ORPO, KTO) across these five axes, using both in-distribution and out-of-distribution datasets. Leveraging a specialized LLM-as-Judge prompt, validated through human studies, we reveal that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity. Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
