面向可扩展离线强化学习的表达性价值学习
Expressive Value Learning for Scalable Offline Reinforcement Learning
AI总结：
针对离线强化学习依赖随时间反向传播或策略蒸馏导致计算昂贵或复合误差的问题，提出EVOR方法，通过流匹配学习正则化Q函数并利用拒绝采样进行推理时策略提取，在多样化任务上优于基线。
摘要：
强化学习（RL）是学习做出一系列决策的强大范式。然而，RL尚未在机器人技术中得到充分利用，主要原因是其缺乏可扩展性。离线RL通过在大型、多样化的数据集上训练智能体，避免了在线RL中昂贵的真实世界交互，提供了一条有前景的途径。将离线RL扩展到日益复杂的数据集需要诸如扩散和流匹配等表达性生成模型。然而，现有方法通常依赖于随时间反向传播（BPTT），这在计算上是极其昂贵的；或者依赖于策略蒸馏，这会引入复合误差并限制了对更大基础策略的可扩展性。在本文中，我们考虑了如何在不依赖蒸馏或随时间反向传播的情况下，开发一种可扩展的离线RL方法的问题。我们引入了用于离线强化学习的表达性价值学习（EVOR）：这是一种可扩展的离线RL方法，它集成了表达性策略和表达性价值函数。EVOR在训练期间通过流匹配学习一个最优的、正则化的Q函数。在推理时，EVOR通过针对表达性价值函数进行拒绝采样来执行推理时策略提取，从而无需重新训练即可实现高效的优化、正则化和计算可扩展的搜索。在经验上，我们证明了EVOR在多样化的离线RL任务集上优于基线，展示了将表达性价值学习集成到离线RL中的优势。
英文摘要：
Reinforcement learning (RL) is a powerful paradigm for learning to make sequences of decisions. However, RL has yet to be fully leveraged in robotics, principally due to its lack of scalability. Offline RL offers a promising avenue by training agents on large, diverse datasets, avoiding the costly real-world interactions of online RL. Scaling offline RL to increasingly complex datasets requires expressive generative models such as diffusion and flow matching. However, existing methods typically depend on either backpropagation through time (BPTT), which is computationally prohibitive, or policy distillation, which introduces compounding errors and limits scalability to larger base policies. In this paper, we consider the question of how to develop a scalable offline RL approach without relying on distillation or backpropagation through time. We introduce Expressive Value Learning for Offline Reinforcement Learning (EVOR): a scalable offline RL approach that integrates both expressive policies and expressive value functions. EVOR learns an optimal, regularized Q-function via flow matching during training. At inference-time, EVOR performs inference-time policy extraction via rejection sampling against the expressive value function, enabling efficient optimization, regularization, and compute-scalable search without retraining. Empirically, we show that EVOR outperforms baselines on a diverse set of offline RL tasks, demonstrating the benefit of integrating expressive value learning into offline RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
