arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08335cs.AI
收藏

谁应得奖励?SHARP:基于Shapley信用的多智能体系统优化

Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System

University of Science and Technology of China(中国科学技术大学)

Yanming Li, Xuelin Zhang, WenJie Lu, Ziye Tang, Maodong Wu, Haotian Luo, Tongtong Wu, Zijie Peng, Hongze Mi, Yibo Feng, Naiqiang Tan, Chao Huang, Lian Peng, Li Shen

AI总结:

针对多智能体系统中信用分配难题,提出SHARP框架,通过分解奖励机制(全局广播奖励、Shapley边际信用奖励和工具过程奖励)实现精确信用归因,显著提升强化学习性能。

摘要:

通过多智能体系统将大型语言模型(LLMs)与外部工具集成,为分解和解决复杂问题提供了一种有前景的新范式。然而,由于信用分配挑战,训练这些系统仍然非常困难,因为通常不清楚哪个特定功能智能体对决策轨迹的成功或失败负责。现有方法通常依赖稀疏或全局广播奖励,无法捕捉个体贡献,导致强化学习效率低下。为解决这些限制,我们引入了基于Shapley的层次化强化策略归因(SHARP),一种通过精确信用归因优化多智能体强化学习的新框架。SHARP通过跨轨迹组归一化智能体特定优势来有效稳定训练,主要通过一种分解奖励机制实现,该机制包括全局广播准确率奖励、每个智能体的基于Shapley的边际信用奖励,以及提高执行效率的工具过程奖励。在各种真实世界基准上的大量实验表明,SHARP显著优于近期最先进的基线,在单智能体和多智能体方法上分别实现了23.66%和14.05%的平均匹配改进。

英文摘要:

Integrating Large Language Models (LLMs) with external tools via multi-agent systems offers a promising new paradigm for decomposing and solving complex problems. However, training these systems remains notoriously difficult due to the credit assignment challenge, as it is often unclear which specific functional agent is responsible for the success or failure of decision trajectories. Existing methods typically rely on sparse or globally broadcast rewards, failing to capture individual contributions and leading to inefficient reinforcement learning. To address these limitations, we introduce the Shapley-based Hierarchical Attribution for Reinforcement Policy (SHARP), a novel framework for optimizing multi-agent reinforcement learning via precise credit attribution. SHARP effectively stabilizes training by normalizing agent-specific advantages across trajectory groups, primarily through a decomposed reward mechanism comprising a global broadcast-accuracy reward, a Shapley-based marginal-credit reward for each agent, and a tool-process reward to improve execution efficiency. Extensive experiments across various real-world benchmarks demonstrate that SHARP significantly outperforms recent state-of-the-art baselines, achieving average match improvements of 23.66% and 14.05% over single-agent and multi-agent approaches, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑