arXiv 2412.112532024-12-15cs.LGcs.AI

表达性模型对于离线强化学习真的必要吗？

Are Expressive Models Truly Necessary for Offline RL?

Tsinghua University（清华大学）；McGill University（麦吉尔大学）；Shanghai AI Laboratory（上海人工智能实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）

Guan Wang, Haoyi Niu, Jianxiong Li, Li Jiang, Jianming Hu, Xianyuan Zhan

备注：: Instead of relying on expressive models, shallow MLPs can also excel in long sequential decision-making tasks with Recursive Skip-Step Planning (RSP)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出递归跳步规划（RSP），用浅层2层MLP通过递归规划子目标实现准确动态建模，在D4RL基准上超越大型表达性模型，达到新SOTA，证明表达性模型并非离线RL必需。

摘要：

在离线强化学习（RL）方法的各个分支中，目标条件监督学习（GCSL）日益受到欢迎，因为它将离线RL问题表述为序列建模任务，从而绕过了传统RL范式中价值学习所面临的众所周知的困难信用分配挑战。然而，序列建模需要在轨迹数据中捕捉跨长时域范围的准确动态，以确保合理的策略性能。为了满足这一要求，利用大型、表达性模型已成为近期文献中的流行选择，但这以显著增加的计算量和推理延迟为代价。矛盾但充满希望的是，我们揭示出，通过采用一种简单的递归规划方案，即便是像浅层2层MLP这样轻量级的模型，也能享有准确的动态一致性，并显著减少相对于大型表达性模型的序列建模误差：该方案基于当前和目标信息递归地规划粗粒度的未来子目标，然后利用从这些子目标真值重新标注的数据学习到的目标条件策略来执行动作。我们将我们的方法命名为递归跳步规划（RSP）。简单而有效，RSP凭借其轻量级结构获得了极大的效率提升，并大幅超越现有方法，在D4RL基准上达到了新的最先进（SOTA）性能，尤其是在多阶段长时域任务中。

英文摘要：

Among various branches of offline reinforcement learning (RL) methods, goal-conditioned supervised learning (GCSL) has gained increasing popularity as it formulates the offline RL problem as a sequential modeling task, therefore bypassing the notoriously difficult credit assignment challenge of value learning in conventional RL paradigm. Sequential modeling, however, requires capturing accurate dynamics across long horizons in trajectory data to ensure reasonable policy performance. To meet this requirement, leveraging large, expressive models has become a popular choice in recent literature, which, however, comes at the cost of significantly increased computation and inference latency. Contradictory yet promising, we reveal that lightweight models as simple as shallow 2-layer MLPs, can also enjoy accurate dynamics consistency and significantly reduced sequential modeling errors against large expressive models by adopting a simple recursive planning scheme: recursively planning coarse-grained future sub-goals based on current and target information, and then executes the action with a goal-conditioned policy learned from data rela-beled with these sub-goal ground truths. We term our method Recursive Skip-Step Planning (RSP). Simple yet effective, RSP enjoys great efficiency improvements thanks to its lightweight structure, and substantially outperforms existing methods, reaching new SOTA performances on the D4RL benchmark, especially in multi-stage long-horizon tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。