离线强化学习中的生存本能
Survival Instinct in Offline Reinforcement Learning
AI总结：
该研究发现离线RL存在“生存本能”现象，即即便用错误奖励训练也能得到安全策略，源于悲观主义与数据偏差的共同作用，并据此提出RL新范式。
摘要：
我们提出了一项关于离线强化学习（RL）算法行为的新观察：在许多基准数据集上，即使使用“错误”的奖励标签（例如处处为零或与真实奖励相反的标签）训练，离线RL也能生成性能良好且安全的策略。这一现象无法用离线RL的回报最大化目标轻易解释。此外，它赋予了离线RL一定程度的鲁棒性，这与其在线RL对应物的特性截然不同——众所周知，在线RL对奖励设计十分敏感。我们证明，这种出人意料的鲁棒性源于离线RL算法中的悲观主义概念与常见数据收集实践中的某些隐式偏差之间的相互作用。正如我们在本研究中所证明的，悲观主义赋予智能体一种“生存本能”，即长期停留在数据支撑集内的动机，而有限且有偏差的数据覆盖范围进一步约束了可行的生存策略集合。形式上，给定一个甚至可能不包含真实奖励的奖励类别，我们确定了训练数据分布的若干条件，这些条件使离线RL能够从该类别内的任意奖励中学习到近似最优且安全的策略。我们认为，在解读现有离线RL基准的结果以及构建未来基准时，应当将生存本能纳入考量。我们的实证与理论结果提出了一种RL新范式：通过有偏差的针对性数据覆盖，引导智能体在奖励不完美的情况下学习到理想行为。
英文摘要：
We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of pessimism in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a "survival instinct", i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class -- which may not even contain the true reward -- we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest a new paradigm for RL, whereby an agent is nudged to learn a desirable behavior with imperfect reward but purposely biased data coverage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
