arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08616cs.LGcs.AI
收藏

突破网格:在大规模离散动作空间中的距离引导强化学习

Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spaces

Technical University of Munich(慕尼黑技术大学);University of Twente(埃因霍温大学)

Heiko Hoppe, Fabian Akkerman, Wouter van Heeswijk, Maximilian Schiffer

AI总结:

本文提出距离引导强化学习(DGRL),通过采样动态邻域和基于距离的更新,在高达10^20个动作的问题中实现高效的强化学习,证明了其在结构任务中的局部价值提升,并在混合连续-离散动作空间中实现泛化。

摘要:

强化学习(RL)越来越多地应用于物流、调度和推荐系统等大规模决策问题,但现有算法在处理大规模离散动作空间时面临维度灾难。我们提出距离引导强化学习(DGRL),结合采样动态邻域和基于距离的更新,使在高达10^20个动作的问题中实现高效的RL。与先前方法不同,DGRL执行随机体积探索,并将策略优化转化为稳定的回归任务,将梯度方差与动作空间基数解耦。在结构化任务中,DGRL证明了局部价值提升。DGRL自然泛化到混合连续-离散动作空间。我们在常规和不规则结构的环境中,相对于最先进的基准,展示了高达66%的性能改进,同时提高了收敛速度和计算复杂度。

英文摘要:

Reinforcement Learning (RL) is increasingly applied to large-scale decision-making problems like logistics, scheduling, and recommender systems, but existing algorithms struggle with the curse of dimensionality in such large discrete action spaces. We propose Distance-Guided Reinforcement Learning (DGRL), combining Sampled Dynamic Neighborhoods and Distance-Based Updates to enable efficient RL in problems with up to $10^{20}$ actions. Unlike prior methods, DGRL performs stochastic volumetric exploration and transforms policy optimization into a stable regression task, decoupling gradient variance from action space cardinality. On structured tasks, DGRL provably guarantees local value improvement. DGRL naturally generalizes to hybrid continuous-discrete action spaces. We demonstrate performance improvements of up to 66% against state-of-the-art benchmarks across regularly and irregularly structured environments, while simultaneously improving convergence speed and computational complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑