arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.06138cs.LG
收藏

基于流匹配的离线强化学习中离散动作的处理

Flow Matching for Offline Reinforcement Learning with Discrete Actions

Department of Computer Science, University of Kentucky(计算机科学系,肯塔基大学);Department of Computing and Information Sciences, Rochester Institute of Technology(计算与信息科学系,罗切斯特理工学院)

Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang, Haibo Yang, Peizhong Ju

AI总结:

本文提出一种适用于离散动作空间的流匹配框架,通过连续时间马尔可夫链和因子化条件路径,扩展了离线强化学习的应用范围,并在多种任务中展示了优越性能。

摘要:

基于扩散模型和流匹配的生成策略在离线强化学习(RL)中显示出强大潜力,但其应用主要局限于连续动作空间。为扩大离线RL的应用范围,我们扩展了流匹配到一个通用框架,支持具有多个目标的离散动作空间。具体来说,我们用连续时间马尔可夫链替代连续流,并使用Q加权流匹配目标进行训练。然后我们将设计扩展到多智能体设置,通过因子化条件路径缓解联合动作空间的指数增长。我们理论证明,在理想条件下,优化此目标可恢复最优策略。广泛实验进一步表明,我们的方法在多样化的设置和基准中表现稳健,包括高维控制、多智能体游戏和动态变化的多目标偏好,同时在实际多模态决策场景中优于传统离线RL方法。我们的离散框架也可通过动作量化应用于连续控制问题,提供代表性和性能之间的灵活权衡。

英文摘要:

Generative policies based on diffusion models and flow matching have shown strong promise for offline reinforcement learning (RL), but their applicability remains largely confined to continuous action spaces. To address a broader range of offline RL settings, we extend flow matching to a general framework that supports discrete action spaces with multiple objectives. Specifically, we replace continuous flows with continuous-time Markov chains, trained using a Q-weighted flow matching objective. We then extend our design to multi-agent settings, mitigating the exponential growth of joint action spaces via a factorized conditional path. For an oracle single policy-improvement step from the behavior distribution, we theoretically show that, under idealized conditions, the Q-weighted objective recovers the KL-regularized Boltzmann policy. Extensive experiments demonstrate that our method performs robustly across diverse settings and benchmarks, including high-dimensional control, multi-agent games, and dynamically changing preferences over multiple objectives, while achieving competitive performance and demonstrating advantages in multimodal decision-making and preference-conditioned policy representation. Our discrete framework can also be applied to continuous-control problems through action quantization, providing a flexible trade-off between representational complexity and performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑