MACCA: 离线多智能体强化学习中的因果信用分配
MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
AI总结：
提出基于动态贝叶斯网络的因果信用分配框架MACCA，通过建模环境变量、状态、动作和奖励的因果关系，实现离线多智能体强化学习中准确且可解释的信用分配。
摘要：
离线多智能体强化学习（MARL）在在线交互不切实际或存在风险的情况下具有重要价值。虽然MARL中的独立学习提供了灵活性和可扩展性，但在离线设置中，由于禁止与环境交互，准确地将信用分配给单个智能体面临挑战。在本文中，我们提出了一种新框架，即多智能体因果信用分配（MACCA），以解决离线MARL设置中的信用分配问题。我们的方法MACCA将生成过程表征为动态贝叶斯网络，捕获环境变量、状态、动作和奖励之间的关系。通过在离线数据上估计该模型，MACCA可以通过分析个体奖励的因果关系来学习每个智能体的贡献，确保准确且可解释的信用分配。此外，我们方法的模块化使其能够无缝集成到各种离线MARL方法中。理论上，我们证明了在离线数据集设置下，底层因果结构和用于生成智能体个体奖励的函数是可识别的，这为我们的建模正确性奠定了基础。在我们的实验中，我们证明MACCA不仅优于最先进的方法，而且在与其他骨干集成时也能提升性能。
英文摘要：
Offline Multi-agent Reinforcement Learning (MARL) is valuable in scenarios where online interaction is impractical or risky. While independent learning in MARL offers flexibility and scalability, accurately assigning credit to individual agents in offline settings poses challenges because interactions with an environment are prohibited. In this paper, we propose a new framework, namely Multi-Agent Causal Credit Assignment (MACCA), to address credit assignment in the offline MARL setting. Our approach, MACCA, characterizing the generative process as a Dynamic Bayesian Network, captures relationships between environmental variables, states, actions, and rewards. Estimating this model on offline data, MACCA can learn each agent's contribution by analyzing the causal relationship of their individual rewards, ensuring accurate and interpretable credit assignment. Additionally, the modularity of our approach allows it to integrate with various offline MARL methods seamlessly. Theoretically, we proved that under the setting of the offline dataset, the underlying causal structure and the function for generating the individual rewards of agents are identifiable, which laid the foundation for the correctness of our modeling. In our experiments, we demonstrate that MACCA not only outperforms state-of-the-art methods but also enhances performance when integrated with other backbones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
