多智能体协作奖励设计以增强强化学习中的推理
Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning
AI总结：
多智能体协作奖励模型通过分解偏好评估和引入集中化聚合器，提升强化学习中的鲁棒性和可解释性，同时提供透明的奖励建模方法。
摘要：
我们提出了CRM（多智能体协作奖励模型），一种框架，它用一组协调的专家评估员取代单一的黑箱奖励模型，以提高RLHF中的鲁棒性和可解释性。传统奖励模型难以同时优化多个有时冲突的偏好维度（例如事实性、有用性和安全性），并且对评分原因的透明度有限。CRM通过将偏好评估分解为领域特定的智能体，每个智能体产生部分信号，以及全局评估器如基于排序的奖励和嵌入相似性奖励来解决这些问题。一个集中化的聚合器在每个时间步融合这些信号，平衡因素如逐步正确性、多智能体一致性以及重复惩罚，产生一个与标准RL流程兼容的单一训练奖励。策略通过优势更新（例如GAE）进行优化，而价值模型回归到聚合奖励，使多视角奖励塑造成为可能，而无需额外的人类注释。为了支持训练和评估，我们引入了rewardBench，一个与CRM协作结构对齐的基准和训练套件。共同，CRM和rewardBench提供了一条实用、模块化的路径，以实现更透明的奖励建模和更稳定的优化。
英文摘要：
We present CRM (Multi-Agent Collaborative Reward Model), a framework that replaces a single black-box reward model with a coordinated team of specialist evaluators to improve robustness and interpretability in RLHF. Conventional reward models struggle to jointly optimize multiple, sometimes conflicting, preference dimensions (e.g., factuality, helpfulness, safety) and offer limited transparency into why a score is assigned. CRM addresses these issues by decomposing preference evaluation into domain-specific agents that each produce partial signals, alongside global evaluators such as ranker-based and embedding-similarity rewards. A centralized aggregator fuses these signals at each timestep, balancing factors like step-wise correctness, multi-agent agreement, and repetition penalties, yielding a single training reward compatible with standard RL pipelines. The policy is optimized with advantage-based updates (e.g., GAE), while a value model regresses to the aggregated reward, enabling multi-perspective reward shaping without requiring additional human annotations beyond those used to train the evaluators. To support training and assessment, we introduce rewardBench, a benchmark and training suite aligned with the collaborative structure of CRM. Together, CRM and rewardBench provide a practical, modular path to more transparent reward modeling and more stable optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
