Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
Zhejiang University(浙江大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院)
Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, Jun Xiao
Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challenges in the training process is credit assignment, which aims to deduce the contributions of each agent according to the global rewards. Existing credit assignment methods focus on either decomposing the joint value function into individual value functions or measuring the impact of local observations and actions on the global value function. These approaches lack a thorough consideration of the complicated interactions among multiple agents, leading to an unsuitable assignment of credit and subsequently mediocre results on MARL. We propose Shapley Counterfactual Credit Assignment, a novel method for explicit credit assignment which accounts for the coalition of agents. Specifically, Shapley Value and its desired properties are leveraged in deep MARL to credit any combinations of agents, which grants us the capability to estimate the individual credit for each agent. Despite this capability, the main technical difficulty lies in the computational complexity of Shapley Value who grows factorially as the number of agents. We instead utilize an approximation method via Monte Carlo sampling, which reduces the sample complexity while maintaining its effectiveness. We evaluate our method on StarCraft II benchmarks across different scenarios. Our method outperforms existing cooperative MARL algorithms significantly and achieves the state-of-the-art, with especially large margins on tasks with more severe difficulties.
Learning What to Skip: Counterfactual Credit Assignment for Efficient Multi-Agent LLM Workflows
The University of British Columbia(不列颠哥伦比亚大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);University of Luxemburg(卢森堡大学);The University of Hong Kong(香港大学);Shenzhen University(深圳大学);Sun Yat-Sen University(中山大学)
STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learning
Renmin University of China(中国人民大学);Peking University(北京大学);King’s College London(伦敦国王学院);National Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能全国重点实验室)
Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agents
Antai College of Economics and Management, Shanghai Jiao Tong University, Shanghai, China(上海交通大学安泰经济管理学院);School of Computer Science and Technology, East China Normal University, Shanghai, China(华东师范大学计算机科学与技术学院);School of Computer Science and Technology, Tongji University, Shanghai, China(同济大学计算机科学与技术学院)
Explaining Reinforcement Learning: A Counterfactual Shapley Values Approach
School of Engineering Mathematics and Technology, University of Bristol(布里斯托大学工程数学与技术学院);School of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院)
Curriculum Learning With Counterfactual Group Relative Policy Advantage For Multi-Agent Reinforcement Learning
School of Information and Communications Engineering, Xi'an Jiaotong University(西安交通大学信息与通信工程学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系)
ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learning
University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学);Jiangxi Science and Technology Normal University(江西科技师范大学);Shenzhen University(深圳大学)
BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents
University of Chicago(芝加哥大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Stanford University(斯坦福大学);University of Science and Technology of China(中国科学技术大学);Meituan(美团)
FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)