通过归一化流实现数据高效的分层目标条件强化学习
Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows
AI总结：
本文提出基于归一化流的分层隐式Q学习框架，通过提升策略表达能力与数据效率，实现更稳健的长周期任务学习。
摘要：
分层目标条件强化学习（H-GCRL）为解决复杂、长周期任务提供了一个强大的框架，通过将其分解为结构化的子目标。然而，其实际应用受到数据效率低和策略表达能力有限的阻碍，尤其是在离线或数据稀缺的环境中。在本文中，提出了一种新的框架，即基于归一化流的分层隐式Q学习（NF-HIQL），该框架在分层的高层和底层都用具有表达能力的归一化流策略替代了单峰高斯策略。这种设计使 log-likelihood 计算变得可行，采样高效，并能够建模丰富的多模式行为。新理论保证包括为实值非体积保持（RealNVP）策略的显式KL散度界以及PAC型样本效率结果，表明NF-HIQL在保持稳定性的同时提高了泛化能力。实验上，NF-HIQL在运动、踢球和多步骤操作等多样化的长周期任务中进行了评估，结果表明NF-HIQL在有限数据下表现出更强的鲁棒性，凸显了基于流的架构在可扩展、数据高效的分层强化学习中的潜力。
英文摘要：
Hierarchical goal-conditioned reinforcement learning (H-GCRL) provides a powerful framework for tackling complex, long-horizon tasks by decomposing them into structured subgoals. However, its practical adoption is hindered by poor data efficiency and limited policy expressivity, especially in offline or data-scarce regimes. In this work, Normalizing flow-based hierarchical implicit Q-learning (NF-HIQL), a novel framework that replaces unimodal gaussian policies with expressive normalizing flow policies at both the high- and low-levels of the hierarchy is introduced. This design enables tractable log-likelihood computation, efficient sampling, and the ability to model rich multimodal behaviors. New theoretical guarantees are derived, including explicit KL-divergence bounds for Real-valued non-volume preserving (RealNVP) policies and PAC-style sample efficiency results, showing that NF-HIQL preserves stability while improving generalization. Empirically, NF-HIQL is evaluted across diverse long-horizon tasks in locomotion, ball-dribbling, and multi-step manipulation from OGBench. NF-HIQL consistently outperforms prior goal-conditioned and hierarchical baselines, demonstrating superior robustness under limited data and highlighting the potential of flow-based architectures for scalable, data-efficient hierarchical reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
