策略无关强化学习：任意类别与骨干网络的离线强化学习及在线强化学习微调
Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
AI总结：
针对现有深度强化学习机制依赖策略类别与骨干网络假设、换用新策略时性能下降的问题，提出策略无关强化学习PA-RL，通过优化动作结合监督损失实现多策略类别的离线训练与在线微调，性能和样本效率最高提升2倍，还首次实现7B通用机器人策略OpenVLA的自主在线微调。
摘要：
决策策略学习领域的最新进展很大程度上归功于对高表达性策略模型的训练，且主要通过模仿学习实现。尽管模仿学习会舍弃非专家数据，但强化学习（RL）仍能从次优数据中学习。然而，对新策略类别开展强化学习训练往往会面临另一项挑战：大多数深度强化学习机制是与策略类别和骨干网络的假设共同开发的，当策略类别发生变化时，性能会大幅下降。例如，软演员-评论家算法（SAC）针对高斯策略采用低方差重参数化策略梯度，但该方法对扩散策略不稳定，且对自回归分类策略不可行。为解决这一问题，我们开发了一种名为策略无关强化学习（PA-RL）的离线强化学习与在线微调方法，能够有效训练具有不同架构和规模的多种策略类别。我们基于的核心思路是：只要将通用监督学习损失应用于“优化后”的动作，它就能替代强化学习中的策略改进步骤。为获取这些优化动作，我们首先从基础策略中采样多个动作，随后通过全局优化（即利用Q函数对多个动作样本重排序）和局部优化（即对单个动作样本执行梯度步骤）来最大化这些候选动作对应的评论家价值。PA-RL 能够完全通过演员-评论家强化学习，对具有自回归 token 或连续动作输出、不同规模的扩散策略和 Transformer 策略进行微调。此外，与现有离线强化学习和在线微调方法相比，PA-RL 的性能和样本效率最高提升了2倍。我们还展示了首个成功利用在线强化学习微调算法 Cal-QL 自主微调 7B 参数通用机器人策略 OpenVLA 的结果，在真实环境中仅用40分钟就将成功率从40%提升至70%。
英文摘要：
Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
