归一化流是连续控制的有效模型
Normalizing Flows are Capable Models for Continuous Control
AI总结：
本文提出一种归一化流架构，可无缝集成到强化学习算法中作为策略、Q函数和占用度量，在模仿学习、离线、目标条件和无监督RL中取得更高性能。
摘要：
现代强化学习（RL）算法通过使用强大的概率模型（如变换器、基于能量的模型以及扩散/流模型）取得了成功。为此，RL研究者通常选择将这些模型融入算法中付出代价——扩散模型表达能力强，但由于依赖求解微分方程而计算密集；自回归变换器模型可扩展，但通常需要学习离散表示。相比之下，归一化流（NFs）似乎提供了一种有吸引力的替代方案，因为它们无需求解微分方程或自回归架构即可实现似然和采样。然而，它们在RL中的潜力受到的关注有限，部分原因是普遍认为归一化流缺乏足够的表达能力。我们证明事实并非如此。基于NFs的最新工作，我们提出了一种单一的NF架构，该架构无缝集成到RL算法中，作为策略、Q函数和占用度量。我们的方法导致了更简单的算法，并在模仿学习、离线、目标条件RL和无监督RL中取得了更高的性能。
英文摘要：
Modern reinforcement learning (RL) algorithms have found success by using powerful probabilistic models, such as transformers, energy-based models, and diffusion/flow-based models. To this end, RL researchers often choose to pay the price of accommodating these models into their algorithms -- diffusion models are expressive, but are computationally intensive due to their reliance on solving differential equations, while autoregressive transformer models are scalable but typically require learning discrete representations. Normalizing flows (NFs), by contrast, seem to provide an appealing alternative, as they enable likelihoods and sampling without solving differential equations or autoregressive architectures. However, their potential in RL has received limited attention, partly due to the prevailing belief that normalizing flows lack sufficient expressivity. We show that this is not the case. Building on recent work in NFs, we propose a single NF architecture which integrates seamlessly into RL algorithms, serving as a policy, Q-function, and occupancy measure. Our approach leads to much simpler algorithms, and achieves higher performance in imitation learning, offline, goal conditioned RL and unsupervised RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
