基于流锚定噪声条件Q学习的离线强化学习：高效且表达力强的方法
Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
AI总结：
提出FAN算法，通过单次流策略迭代和单高斯噪声样本实现高效离线强化学习，在保持高性能的同时显著降低计算成本。
摘要：
我们提出流锚定噪声条件Q学习（FAN），一种高效且高性能的离线强化学习算法。近期工作表明，表达力强的流策略和分布性评论家能提升离线强化学习性能，但计算成本高。具体而言，流策略需要迭代采样才能产生单个动作，分布性评论家需要计算多个样本（如分位数）来估计价值。为解决这些低效问题并保持高性能，我们引入FAN。我们的方法采用行为正则化技术，仅需单次流策略迭代，且分布性评论家仅需单个高斯噪声样本。我们对收敛性和性能边界的理论分析表明，这些简化不仅提高了效率，还带来了更优的任务性能。在机器人操作和运动任务上的实验表明，FAN实现了最先进的性能，同时显著减少了训练和推理时间。我们在https://github.com/brianlsy98/FAN 发布代码。
英文摘要：
We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
