arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.28281cs.LG
收藏

从人类反馈中鲁棒的离线多智能体强化学习

Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback

MPI-SWS(马克斯·普朗克软件系统研究所);University of Warwick(华威大学);Visa(Visa公司)

Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Adish Singla, Goran Radanović

AI总结:

本文研究了在强污染模型下,从人类反馈中进行离线多智能体强化学习(MARLHF)的鲁棒性问题,提出了一种鲁棒估计器和基于粗相关均衡的算法,解决了计算复杂性问题。

摘要:

我们考虑在离线多智能体强化学习从人类反馈(MARLHF)中对数据污染的鲁棒性,采用强污染模型:给定一个轨迹偏好元组的数据集D(每个偏好是一个n维二进制标签向量,表示每个n智能体的偏好),ε比例的样本可能被任意污染。我们使用线性马尔可夫游戏框架建模该问题。首先,在统一覆盖假设下(即每个感兴趣的策略在干净(污染前)数据中充分表示),我们引入一种鲁棒估计器,保证纳什均衡差距的O(ε^{1 - o(1)})界。接着,我们转向更具挑战性的单方面覆盖设置,其中仅纳什均衡及其单人偏离被覆盖。在此情况下,我们提出的算法实现了纳什差距的O(√ε)界。然而,这两种过程都面临计算不可行的问题。为了解决这个问题,我们将解决方案概念放松为粗相关均衡(CCE)。在相同的单方面覆盖制度下,我们推导出一个准多项式时间算法,其CCE差距的规模为O(√ε)。据我们所知,这是首次系统地处理对抗性数据污染在离线MARLHF中的问题。

英文摘要:

We consider robustness against data corruption in offline multi-agent reinforcement learning from human feedback (MARLHF) under a strong-contamination model: given a dataset $D$ of trajectory-preference tuples (each preference being an $n$-dimensional binary label vector representing each of the $n$ agents' preferences), an $ε$-fraction of the samples may be arbitrarily corrupted. We model the problem using the framework of linear Markov games. First, under a uniform coverage assumption - where every policy of interest is sufficiently represented in the clean (prior to corruption) data - we introduce a robust estimator that guarantees an $O(ε^{1 - o(1)})$ bound on the Nash equilibrium gap. Next, we move to the more challenging unilateral coverage setting, in which only a Nash equilibrium and its single-player deviations are covered. In this case, our proposed algorithm achieves an $O(\sqrtε)$ bound on the Nash gap. Both of these procedures, however, suffer from intractable computation. To address this, we relax our solution concept to coarse correlated equilibria (CCE). Under the same unilateral coverage regime, we derive a quasi-polynomial-time algorithm whose CCE gap scales as $O(\sqrtε)$. To the best of our knowledge, this is the first systematic treatment of adversarial data corruption in offline MARLHF.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑