价值学习真的是离线强化学习的主要瓶颈吗？
Is Value Learning Really the Main Bottleneck in Offline RL?
AI总结：
该研究针对离线RL性能常不及模仿学习的问题，通过实证分析发现策略提取算法选择和测试时策略泛化才是主要瓶颈，而非普遍认为的价值学习，并提出两种测试时策略改进方法提升性能。
摘要：
尽管模仿学习需要获取高质量数据，但离线强化学习（RL）原则上应能通过使用价值函数，在数据质量低得多的情况下取得相近甚至更优的性能。然而，当前研究结果表明，离线RL的表现往往不如模仿学习，且人们通常不清楚究竟是什么制约了离线RL的性能。受这一观察启发，我们旨在厘清当前离线RL算法中的瓶颈所在。虽然离线RL性能不佳通常被归咎于价值函数不完善，但我们提出疑问：离线RL的主要瓶颈确实在于价值函数学习，还是另有其他因素？为回答这一问题，我们针对离线RL问题中的三个方面开展了系统性实证研究：（1）价值学习、（2）策略提取、（3）策略泛化，分析了这些组成部分如何影响性能。我们得出了两个出人意料的发现。首先，我们发现策略提取算法的选择会显著影响离线RL的性能和可扩展性，其影响程度往往超过价值学习目标。例如，我们证明常见的价值加权行为克隆目标（如AWR，优势加权回归）无法充分利用已学习的价值函数，而改用行为约束策略梯度目标（如DDPG+BC，深度确定性策略梯度结合行为克隆）通常能大幅提升性能和可扩展性。其次，我们发现提升离线RL性能的一大障碍往往是策略在训练数据支撑集之外的测试时状态上的泛化能力不足，而非策略在分布内状态上的学习效果。随后我们证明，使用次优但覆盖度高的数据或测试时策略训练技术能够在实际中解决这一泛化问题。具体而言，我们提出了两种简单的测试时策略改进方法，并证明这些方法能带来更优的性能。
英文摘要：
While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
