arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01361cs.LG
收藏

到极限:强化学习中奖励的重塑

To the Max: Reinventing Reward in Reinforcement Learning

Delft University of Technology(代尔夫特理工大学)

Grigorii Veviurko, Wendelin Böhmer, Mathijs de Weerdt

AI总结:

本文提出最大奖励强化学习(max-reward RL),优化最大而非累积奖励,适用于确定性与随机环境,在目标到达任务中优于标准RL。

摘要:

在强化学习(RL)中,不同的奖励函数可以定义相同的最优策略,但会导致截然不同的学习性能。对于某些奖励函数,智能体可能陷入次优行为,而对于其他奖励函数,它则能高效地解决任务。因此,选择一个好的奖励函数是一个极其重要且具有挑战性的问题。在本文中,我们探索了一种使用奖励进行学习的替代方法。我们引入了“最大奖励强化学习”(max-reward RL),其中智能体优化的是最大奖励而非累积奖励。与早期工作不同,我们的方法适用于确定性和随机环境,并且可以轻松地与最先进的RL算法结合。在实验中,我们研究了最大奖励RL算法在Gymnasium-Robotics中两个目标到达环境中的性能,并展示了其相对于标准RL的优势。代码可在https://github.com/veviurko/To-the-Max获取。

英文摘要:

In reinforcement learning (RL), different reward functions can define the same optimal policy but result in drastically different learning performance. For some, the agent gets stuck with a suboptimal behavior, and for others, it solves the task efficiently. Choosing a good reward function is hence an extremely important yet challenging problem. In this paper, we explore an alternative approach for using rewards for learning. We introduce \textit{max-reward RL}, where an agent optimizes the maximum rather than the cumulative reward. Unlike earlier works, our approach works for deterministic and stochastic environments and can be easily combined with state-of-the-art RL algorithms. In the experiments, we study the performance of max-reward RL algorithms in two goal-reaching environments from Gymnasium-Robotics and demonstrate its benefits over standard RL. The code is available at https://github.com/veviurko/To-the-Max.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑