arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.08360cs.LGeess.SP
收藏

数字表亲选择的覆盖分析——改进多环境Q学习

Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning

University of Southern California(南加州大学);University of California at San Diego(加州大学圣地亚哥分校)

Talha Bozkus, Tara Javidi, Urbashi Mitra

AI总结:

本文对多环境混合Q学习(MEMQ)算法进行概率覆盖分析,推导了覆盖系数的上下界,并提出一种基于覆盖系数的新型MEMQ算法,显著降低了平均策略误差并提升了运行速度。

摘要:

Q学习被广泛用于优化具有未知系统动态的各种大维网络。最新进展包括多环境混合Q学习(MEMQ)算法,该算法在多个结构相关但不同的环境中利用多个独立的Q学习算法,并在准确率、复杂度和鲁棒性方面优于几种最先进的Q学习算法。我们在此进行全面的概率覆盖分析,以确保MEMQ算法的最佳数据覆盖条件。首先,我们推导了MEMQ算法不同覆盖系数(CC)的期望和方差的上界和下界。利用这些界限,我们开发了一种简单的方法来比较MEMQ算法中多个环境的效用。与我们之前提出的偏序方法相比,这种方法似乎是接近最优的。我们还提出了一种基于CC的新型MEMQ算法,以提高现有MEMQ算法的准确率和复杂度。使用具有四种不同图属性的随机网络图进行了数值实验。与偏序方法相比,我们的算法可将平均策略误差(APE)降低65%,且比穷举搜索快95%。与几种最先进的强化学习和先前的MEMQ算法相比,它还实现了60%的APE降低。此外,我们在数值上验证了理论结果,并展示了其随动作空间大小的可扩展性。

英文摘要:

Q-learning is widely employed for optimizing various large-dimensional networks with unknown system dynamics. Recent advancements include multi-environment mixed Q-learning (MEMQ) algorithms, which utilize multiple independent Q-learning algorithms across multiple, structurally related but distinct environments and outperform several state-of-the-art Q-learning algorithms in terms of accuracy, complexity, and robustness. We herein conduct a comprehensive probabilistic coverage analysis to ensure optimal data coverage conditions for MEMQ algorithms. First, we derive upper and lower bounds on the expectation and variance of different coverage coefficients (CC) for MEMQ algorithms. Leveraging these bounds, we develop a simple way of comparing the utilities of multiple environments in MEMQ algorithms. This approach appears to be near optimal versus our previously proposed partial ordering approach. We also present a novel CC-based MEMQ algorithm to improve the accuracy and complexity of existing MEMQ algorithms. Numerical experiments are conducted using random network graphs with four different graph properties. Our algorithm can reduce the average policy error (APE) by 65% compared to partial ordering and is 95% faster than the exhaustive search. It also achieves 60% less APE than several state-of-the-art reinforcement learning and prior MEMQ algorithms. Additionally, we numerically verify the theoretical results and show their scalability with the action-space size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑