arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.07211cs.LG
收藏

提高无线通信资源分配中深度强化学习的训练效率:生成扩散模型的作用

Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Models

Internet of Things Thrust at the Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)物联网部)

Xinren Zhang, Jiadong Yu

AI总结:

针对移动无线网络动态资源分配中DRL预训练策略易失效、重训练成本高的问题,提出D2RL框架,利用GDMs优化状态空间、动作探索和奖励设计,提升训练效率并保持策略性能。

摘要:

移动无线网络中的动态资源分配涉及复杂、时变的优化问题,促使人们采用深度强化学习(DRL)。然而,大多数现有研究依赖预训练策略,忽视了会使策略迅速失效的动态环境变化。定期重训练不可避免,但会产生高昂的计算成本和能耗,这对资源受限的无线系统而言是关键问题。我们确定了低效重训练的三个根本原因:高维状态空间、动作空间探索-利用权衡的次优性以及奖励设计的局限性。为克服这些限制,我们提出基于扩散的深度强化学习(D2RL),利用生成扩散模型(GDMs)全面增强DRL的三个组件。GDMs的迭代细化过程和分布建模能够(1)生成多样化的状态样本以提升环境理解,(2)平衡动作空间探索以逃离局部最优,(3)设计更优的判别性奖励函数以更好地评估动作质量。我们的框架有两种模式:模式I利用GDMs探索奖励空间并设计严格评估动作质量的判别性奖励函数;模式II合成多样化状态样本以增强环境理解和泛化能力。大量实验表明,在无线通信资源分配任务中,D2RL相比传统DRL方法实现了更快的收敛速度和更低的计算成本,同时保持了有竞争力的策略性能。这项工作强调了GDMs在克服无线网络DRL训练基本瓶颈方面的变革潜力,为实际实时部署铺平了道路。

英文摘要:

Dynamic resource allocation in mobile wireless networks involves complex, time-varying optimization problems, motivating the adoption of deep reinforcement learning (DRL). However, most existing works rely on pre-trained policies, overlooking dynamic environmental changes that rapidly invalidate the policies. Periodic retraining becomes inevitable but incurs prohibitive computational costs and energy consumption-critical concerns for resource-constrained wireless systems. We identify three root causes of inefficient retraining: high-dimensional state spaces, suboptimal action spaces exploration-exploitation trade-offs, and reward design limitations. To overcome these limitations, we propose Diffusion-based Deep Reinforcement Learning (D2RL), which leverages generative diffusion models (GDMs) to holistically enhance all three DRL components. Iterative refinement process and distribution modelling of GDMs enable (1) the generation of diverse state samples to improve environmental understanding, (2) balanced action space exploration to escape local optima, and (3) the design of discriminative reward functions that better evaluate action quality. Our framework operates in two modes: Mode I leverages GDMs to explore reward spaces and design discriminative reward functions that rigorously evaluate action quality, while Mode II synthesizes diverse state samples to enhance environmental understanding and generalization. Extensive experiments demonstrate that D2RL achieves faster convergence and reduced computational costs over conventional DRL methods for resource allocation in wireless communications while maintaining competitive policy performance. This work underscores the transformative potential of GDMs in overcoming fundamental DRL training bottlenecks for wireless networks, paving the way for practical, real-time deployments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑