策略性保守Q学习
Strategically Conservative Q-Learning
AI总结：
本文提出策略性保守Q学习（SCQ），通过区分易估与难估的分布外数据，在保持价值估计悲观的同时减少过度保守，并在D4RL基准上取得优于现有方法的性能。
摘要：
离线强化学习（RL）是一种极具吸引力的范式，它通过利用预先收集的静态数据集来扩展RL的实际应用，从而避免与收集在线交互相关的限制。离线RL的主要困难在于，当遇到分布外（OOD）动作时，如何减轻近似误差的影响；如果处理不当，将导致策略偏好OOD动作，进而可能引发意外甚至灾难性的结果。尽管已有多种工作被提出来解决这一问题，但它们往往在OOD区域及其周围过度抑制价值函数，导致价值估计过于悲观。在本文中，我们提出了一种名为策略性保守Q学习（SCQ）的新框架，该框架区分了易于估计和难以估计的OOD数据，最终得到不那么保守的价值估计。我们的方法利用神经网络固有的插值优势，同时谨慎应对其在外推方面的局限性，以获得悲观但仍经过适当校准的价值估计。理论分析还表明，SCQ学习到的价值函数仍然是保守的，但可能远不如保守Q学习（CQL）那样保守。最后，在D4RL基准任务上的广泛评估表明，我们提出的方法优于最先进的方法。我们的代码可通过\url{https://github.com/purewater0901/SCQ}获取。
英文摘要：
Offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility by leveraging pre-collected, static datasets, thereby avoiding the limitations associated with collecting online interactions. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions; doing so ineffectively will lead to policies that prefer OOD actions, which can lead to unexpected and potentially catastrophic results. Despite the variety of works proposed to address this issue, they tend to excessively suppress the value function in and around OOD regions, resulting in overly pessimistic value estimates. In this paper, we propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate, ultimately resulting in less conservative value estimates. Our approach exploits the inherent strengths of neural networks to interpolate, while carefully navigating their limitations in extrapolation, to obtain pessimistic yet still property calibrated value estimates. Theoretical analysis also shows that the value function learned by SCQ is still conservative, but potentially much less so than that of Conservative Q-learning (CQL). Finally, extensive evaluation on the D4RL benchmark tasks shows our proposed method outperforms state-of-the-art methods. Our code is available through \url{https://github.com/purewater0901/SCQ}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
