SAC Flow: 通过速度重参数化序列建模高效训练基于流的策略
SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
AI总结：
SAC Flow通过速度重参数化序列建模，高效训练基于流的策略，实现连续控制和机器人操作的高性能表现。
摘要：
训练具有表达能力的基于流的策略时，使用非策略强化学习往往因多步动作采样过程中的梯度病态而不稳定。我们追溯这种不稳定性到一个根本联系：流回放在代数上等同于残差递归计算，使其容易受到与RNN相同的消失和爆炸梯度问题的影响。为了解决这个问题，我们利用现代序列模型的原则重新参数化速度网络，引入了两种稳定的架构：Flow-G，其包含一个门控速度，以及Flow-T，其利用解码速度。然后我们开发了一种实用的SAC基于算法，通过噪声增强的回放，使这些策略能够直接端到端地训练。我们的方法支持从头开始和离线到在线学习，并在连续控制和机器人操作基准上实现了最先进的性能，消除了诸如策略蒸馏或替代目标等常见工作方式的需要。
英文摘要：
Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process. We trace this instability to a fundamental connection: the flow rollout is algebraically equivalent to a residual recurrent computation, making it susceptible to the same vanishing and exploding gradients as RNNs. To address this, we reparameterize the velocity network using principles from modern sequential models, introducing two stable architectures: Flow-G, which incorporates a gated velocity, and Flow-T, which utilizes a decoded velocity. We then develop a practical SAC-based algorithm, enabled by a noise-augmented rollout, that facilitates direct end-to-end training of these policies. Our approach supports both from-scratch and offline-to-online learning and achieves state-of-the-art performance on continuous control and robotic manipulation benchmarks, eliminating the need for common workarounds like policy distillation or surrogate objectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
