ARCANE:一种多智能体框架,用于可解释和可配置的对齐
ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment
AI总结:
ARCANE是一种多智能体框架,通过动态规则生成实现可解释和可配置的对齐,适用于复杂长期任务。
摘要:
随着基于大语言模型的智能体被越来越多地部署到长期任务中,保持其与利益相关者偏好的对齐变得至关重要。在这样的情况下,有效的对齐需要可解释的奖励模型,以便利益相关者可以理解和审计模型目标。此外,奖励模型必须能够在交互时间引导智能体,允许偏好变化被纳入其中而无需重新训练。我们介绍了ARCANE,一种将对齐视为多智能体协作问题的框架,该框架动态地将利益相关者的偏好表示为自然语言的规则:可验证的标准的加权集合,可以从任务上下文中即时生成。受效用理论的启发,我们将规则学习制定为重建问题,并应用一种正则化的组序列策略优化(GSPO)程序,以平衡可解释性、忠实度和计算效率。使用从GDPVal基准中提取的219个标记的规则库,我们在需要多步推理和工具使用的挑战性任务上评估了ARCANE。学习到的规则产生紧凑、清晰的评估,并允许在不重新训练的情况下配置可调节的权衡(例如,正确性与简洁性)。我们的结果表明,基于规则的奖励模型为复杂、长期的人工智能系统提供了可解释、测试时间自适应对齐的有希望的路径。
英文摘要:
As agents based on large language models are increasingly deployed to long-horizon tasks, maintaining their alignment with stakeholder preferences becomes critical. Effective alignment in such settings requires reward models that are interpretable so that stakeholders can understand and audit model objectives. Moreover, reward models must be capable of steering agents at interaction time, allowing preference shifts to be incorporated without retraining. We introduce ARCANE, a framework that frames alignment as a multi-agent collaboration problem that dynamically represents stakeholder preferences as natural-language rubrics: weighted sets of verifiable criteria that can be generated on-the-fly from task context. Inspired by utility theory, we formulate rubric learning as a reconstruction problem and apply a regularized Group-Sequence Policy Optimization (GSPO) procedure that balances interpretability, faithfulness, and computational efficiency. Using a corpus of 219 labeled rubrics derived from the GDPVal benchmark, we evaluate ARCANE on challenging tasks requiring multi-step reasoning and tool use. The learned rubrics produce compact, legible evaluations and enable configurable trade-offs (e.g., correctness vs. conciseness) without retraining. Our results show that rubric-based reward models offer a promising path toward interpretable, test-time adaptive alignment for complex, long-horizon AI systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。