动态对齐与集体代理：迈向一个可扩展的自我改进框架以实现开放式的LLM对齐
Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
AI总结：
本文提出动态对齐框架，通过集体代理价值实现LLM的自我改进和可扩展对齐。
摘要：
大型语言模型（LLMs）通常通过偏好数据或预定义的原则如有用性、诚实和无害性来对齐人类价值观。然而，随着人工智能系统向人工通用智能（AGI）和人工超智能（ASI）发展，此类价值观系统可能变得不足。此外，基于人类反馈的对齐仍然耗费资源且难以扩展。尽管AI反馈基于的自我改进对齐方法已被探索作为可扩展的替代方案，但它们大多仍局限于传统对齐价值观。在本工作中，我们探索了更全面的对齐目标和可扩展、自我改进的对齐方法。旨在超越传统对齐规范，我们引入了集体代理（CA）——一个统一且开放的对齐价值，鼓励集成的代理能力。我们还提出了动态对齐——一种使LLM能够迭代对齐自身的对齐框架。动态对齐包含两个关键组成部分：（1）使用LLM自动生成训练数据集，以及（2）一种自我奖励机制，其中策略模型评估其自身的输出候选并为基于GRPO的学习分配奖励。实验结果表明，我们的方法成功地将模型对齐到CA，同时保持了通用NLP能力。
英文摘要：
Large Language Models (LLMs) are typically aligned with human values using preference data or predefined principles such as helpfulness, honesty, and harmlessness. However, as AI systems progress toward Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI), such value systems may become insufficient. In addition, human feedback-based alignment remains resource-intensive and difficult to scale. While AI-feedback-based self-improving alignment methods have been explored as a scalable alternative, they have largely remained constrained to conventional alignment values. In this work, we explore both a more holistic alignment objective and a scalable, self-improving alignment approach. Aiming to transcend conventional alignment norms, we introduce Collective Agency (CA)-a unified and open-ended alignment value that encourages integrated agentic capabilities. We also propose Dynamic Alignment-an alignment framework that enables an LLM to iteratively align itself. Dynamic Alignment comprises two key components: (1) automated training dataset generation with LLMs, and (2) a self-rewarding mechanism, where the policy model evaluates its own output candidates and assigns rewards for GRPO-based learning. Experimental results demonstrate that our approach successfully aligns the model to CA while preserving general NLP capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
