Q-greedyUCB:一种新的自适应资源有效调度探索策略
Q-greedyUCB: a New Exploration Policy for Adaptive and Resource-efficient Scheduling
摘要:
本文提出了一种学习算法,以找到一个调度策略,实现最佳的延迟功率权衡通信系统。强化学习(RL)用于最小化给定能量约束的预期延迟,其中环境(如流量到达率或信道条件)可能随时间而变化。为此,该问题被制定为一个无限时域马尔可夫决策过程(MDP)的约束。在处理约束优化问题时,采用拉格朗日松弛技术进行求解,然后提出了一种Q-学习的变体Q-greedyUCB,它结合了Q-学习的\emph{average}奖励算法和上置信限(UCB)策略来解决这一决策问题。通过数学分析证明了Q-greedyUCB算法的收敛性。仿真结果表明,Q-greedyUCB找到了最优调度策略,并且在累积奖励方面(即,延迟和能量的加权和)和收敛速度。我们还表明,我们的算法可以减少高达12%的遗憾相比,Q学习与$\varepsilon$ -贪婪和平均回报RL算法。
英文摘要:
This paper proposes a learning algorithm to find a scheduling policy that achieves an optimal delay-power trade-off in communication systems. Reinforcement learning (RL) is used to minimize the expected latency for a given energy constraint where the environments such as traffic arrival rates or channel conditions can change over time. For this purpose, this problem is formulated as an infinite-horizon Markov Decision Process (MDP) with constraints. To handle the constrained optimization problem, we adopt the Lagrangian relaxation technique to solve it. Then, we propose a variant of Q-learning, Q-greedyUCB that combines Q-learning for \emph{average} reward algorithm and Upper Confidence Bound (UCB) policy to solve this decision-making problem. We prove that the Q-greedyUCB algorithm is convergent through mathematical analysis. Simulation results show that Q-greedyUCB finds an optimal scheduling strategy, and is more efficient than Q-learning with the $\varepsilon$-greedy and Average-payoff RL algorithm in terms of the cumulative reward (i.e., the weighted sum of delay and energy) and the convergence speed. We also show that our algorithm can reduce the regret by up to 12% compared to the Q-learning with the $\varepsilon$-greedy and Average-payoff RL algorithm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。