arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.06286cs.LGcs.AI
收藏

通过对抗行为抑制 Q 学习中的过高估计

Suppressing Overestimation in Q-Learning through Adversarial Behaviors

KAIST(韩国科学技术院)

HyeAnn Lee, Donghwan Lee

AI总结:

本文提出虚拟对抗 Q 学习(DAQ),通过引入虚拟对抗玩家将学习建模为双人零和博弈,以抑制 Q 学习中的过高估计偏差,并统一了多种 Q 学习变体,实验验证了其收敛性和性能提升。

摘要:

本文的目标是提出一种带有虚拟对抗玩家的新 Q 学习算法,称为虚拟对抗 Q 学习(DAQ),该算法能够有效调控标准 Q 学习中的过高估计偏差。引入虚拟玩家后,学习过程可以被形式化为一个双人零和博弈。所提出的 DAQ 在单一框架中统一了多种用于控制过高估计偏差的 Q 学习变体,例如 maxmin Q 学习和 minmax Q 学习(本文提出)。所提出的 DAQ 是一种简单而有效的方法,通过虚拟对抗行为来抑制过高估计偏差,并且可以方便地应用于现成的强化学习算法以提升性能。本文通过适配对抗 Q 学习,从综合视角分析了 DAQ 的有限时间收敛性。所提出的 DAQ 的性能在多种基准环境中得到了实证验证。

英文摘要:

The goal of this paper is to propose a new Q-learning algorithm with a dummy adversarial player, which is called dummy adversarial Q-learning (DAQ), that can effectively regulate the overestimation bias in standard Q-learning. With the dummy player, the learning can be formulated as a two-player zero-sum game. The proposed DAQ unifies several Q-learning variations to control overestimation biases, such as maxmin Q-learning and minmax Q-learning (proposed in this paper) in a single framework. The proposed DAQ is a simple but effective way to suppress the overestimation bias thourgh dummy adversarial behaviors and can be easily applied to off-the-shelf reinforcement learning algorithms to improve the performances. A finite-time convergence of DAQ is analyzed from an integrated perspective by adapting an adversarial Q-learning. The performance of the suggested DAQ is empirically demonstrated under various benchmark environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑