B3C: 一种针对离线多智能体强化学习的极简方法
B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning
AI总结：
B3C通过引入批评者剪裁和非线性价值分解，有效解决离线多智能体强化学习中的过估计问题，提升性能。
摘要：
在离线强化学习（RL）中，由于在策略评估期间选择未见过的动作而产生的过高估计是一个主要挑战。在单智能体设置中，一种极简方法——将行为克隆（BC）正则化添加到现有的在线RL算法中——已被证明是有效的；然而，这种方法在多智能体设置中研究不足。特别是，由于存在多个动作，过估计在多智能体设置中变得更糟，导致基于BC正则化的做法容易遭受过正则化或批评者发散的问题。为了解决这个问题，我们提出了一种简单而有效的方法，即带有批评者剪裁的行为克隆正则化（B3C），它基于数据集中的最大回报剪裁目标批评者值，并推动RL目标上的权重限制超过BC正则化，从而提高性能。此外，我们利用现有的价值分解技术，特别是非线性分解，这在离线设置中研究不足。结合非线性价值分解，B3C在各种离线多智能体基准上优于最先进的算法。
英文摘要：
Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC) regularization to existing online RL algorithms -- has been shown to be effective; however, this approach is understudied in multi-agent settings. In particular, overestimation becomes worse in multi-agent settings due to the presence of multiple actions, resulting in the BC regularization-based approach easily suffering from either over-regularization or critic divergence. To address this, we propose a simple yet effective method, Behavior Cloning regularization with Critic Clipping (B3C), which clips the target critic value in policy evaluation based on the maximum return in the dataset and pushes the limit of the weight on the RL objective over BC regularization, thereby improving performance. Additionally, we leverage existing value factorization techniques, particularly non-linear factorization, which is understudied in offline settings. Integrated with non-linear value factorization, B3C outperforms state-of-the-art algorithms on various offline multi-agent benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
