arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09068cs.LGcs.AI
收藏

通过标准化基线与评估消除离线多智能体强化学习(MARL)中的进展假象

Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation

InstaDeep(英斯塔迪普公司);University of Cape Town(开普敦大学);INRS, Montreal(蒙特利尔国家科学研究院);NITheCS, Stellenbosch(斯泰伦博斯国家理论科学研究所)

Claude Formanek, Callum Rhys Tilbury, Louise Beyers, Jonathan Shock, Arnu Pretorius

AI总结:

该研究指出离线MARL领域现有评估与基线的不一致问题,证明简单精良基线在近75%的数据集上达到或超当前SOTA,提出标准化评估方法与基线以提升领域实证严谨性。

摘要:

离线多智能体强化学习(MARL)是一个在现实应用中极具前景的新兴领域。遗憾的是,当前离线MARL研究受困于基线与评估协议的不一致性,这最终导致研究人员难以准确评估进展、信任新提出的创新成果,也难以便捷地在前人工作基础上开展研究。本文中,我们首先通过对已发表的离线MARL工作开展代表性研究,指出现有新型算法性能衡量方法存在的重大缺陷。其次,通过与这些前人工作直接对比,我们证明实现精良的简单基线能在广泛任务上达到当前最优(SOTA)结果。具体而言,我们发现在前人工作使用的47个数据集中的35个(近75%的案例)上,我们的基线性能与当前所谓的SOTA相当甚至更优。值得注意的是,我们的基线通常大幅优于这些更复杂的算法。最后,我们引入了一套简洁的标准化评估方法,并提供了在多种场景下具备统计稳健性的基线实现结果,可用于未来工作的对比,以此修正前人工作中暴露的缺陷。我们的方案包含易于采纳的简单合理步骤,结合扎实的基线与对比结果,能显著提升未来离线MARL领域实证研究的整体严谨性。

英文摘要:

Offline multi-agent reinforcement learning (MARL) is an emerging field with great promise for real-world applications. Unfortunately, the current state of research in offline MARL is plagued by inconsistencies in baselines and evaluation protocols, which ultimately makes it difficult to accurately assess progress, trust newly proposed innovations, and allow researchers to easily build upon prior work. In this paper, we firstly identify significant shortcomings in existing methodologies for measuring the performance of novel algorithms through a representative study of published offline MARL work. Secondly, by directly comparing to this prior work, we demonstrate that simple, well-implemented baselines can achieve state-of-the-art (SOTA) results across a wide range of tasks. Specifically, we show that on 35 out of 47 datasets used in prior work (almost 75% of cases), we match or surpass the performance of the current purported SOTA. Strikingly, our baselines often substantially outperform these more sophisticated algorithms. Finally, we correct for the shortcomings highlighted from this prior work by introducing a straightforward standardised methodology for evaluation and by providing our baseline implementations with statistically robust results across several scenarios, useful for comparisons in future work. Our proposal includes simple and sensible steps that are easy to adopt, which in combination with solid baselines and comparative results, could substantially improve the overall rigour of empirical science in offline MARL moving forward.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑