熵正则化对偶匹配用于离线强化学习
Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning
AI总结：
本文提出最大熵对偶匹配框架，通过镜像下降熵最大化和混合行为先验机制，缓解流行偏差和支撑绑定问题，提升离线数据中最优策略的提取能力。
摘要：
将具有表达力的生成策略，如流匹配模型，整合到离线强化学习（RL）中，使智能体能够捕捉复杂多模式行为。虽然通过连续共轭方法的Q学习与对偶匹配（QAM）通过连续共轭方法稳定了策略优化，但其本质上仍受限于固定行为分布。这种依赖性导致了流行偏差，可能抑制低密度区域中的高奖励动作，并产生支撑绑定，限制非流形探索。现有解决方案，如附加残差高斯策略，往往重新引入与单峰分布相关的表达性瓶颈。在本文中，我们提出最大熵对偶匹配（ME-AM），一个统一框架，该框架在连续流公式化中解决了这些限制。ME-AM包含两个机制：（1）一个镜像下降熵最大化目标，缓解流行偏差以促进从离线数据集中提取最优策略；（2）一个混合行为先验，扩展几何支持以涵盖非分布高奖励区域。通过探索此扩展几何，ME-AM识别出稳健的动作，同时保持生成向量场的绝对连续性。实证上，ME-AM在多样化的稀疏奖励连续控制环境中表现出竞争力或优于现有最先进（SOTA）方法的性能。
英文摘要：
Integrating expressive generative policies, such as flow-matching models, into offline reinforcement learning (RL) allows agents to capture complex, multi-modal behaviors. While Q-learning with Adjoint Matching (QAM) stabilizes policy optimization via the continuous adjoint method, it remains inherently bound to the fixed behavior distribution. This dependence induces a \textit{popularity bias} that can suppress high-reward actions in low-density regions, and creates a \textit{support binding} that restricts off-manifold exploration. Existing workarounds, such as appending \textit{residual} Gaussian policies, often re-introduce the expressivity bottlenecks associated with unimodal distributions. In this work, we propose \textit{Maximum Entropy Adjoint Matching} (ME-AM), a unified framework that addresses these limitations within the continuous flow formulation. ME-AM incorporates two mechanisms: (1) a Mirror Descent entropy maximization objective that mitigates the popularity bias to facilitate the extraction of optimal policies from offline datasets, and (2) a \textit{Mixture Behavior Prior} that broadens the geometric support to encompass out-of-distribution high-reward regions. By exploring this extended geometry, ME-AM identifies robust actions while preserving the absolute continuity of the generative vector field. Empirically, ME-AM demonstrates competitive or superior performance compared to prior state-of-the-art (SOTA) methods across a diverse suite of sparse-reward continuous control environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
