无限维系统的强化学习
Reinforcement Learning for Infinite-Dimensional Systems
AI总结:
针对大规模智能体系统RL计算成本高且性能下降的问题,提出基于无限维函数空间参数化模型的分层算法,利用矩核变换与过滤结构实现最优策略学习并快速收敛。
摘要:
近年来,各个科学领域对由与异构环境交互的大量智能体群体组成的大规模系统进行强化学习(RL)的兴趣显著激增。然而,这些系统的大规模特性往往会导致大多数最先进的RL技术面临高计算成本或性能下降的问题。为了应对这些挑战,我们提出了一种新颖的RL架构,并推导出有效的算法来学习任意大型智能体系统的最优策略。在我们的公式中,我们将此类系统建模为定义在无限维函数空间上的参数化控制系统。然后,我们开发了一种矩核变换,将参数化系统和价值函数映射到再生核Hilbert空间中。这种变换为RL问题生成了一系列有限维矩表示,并组织成一种过滤结构。利用这种RL过滤,我们开发了一种分层算法,用于学习无限维参数化系统的最优策略。为了提高算法的效率,我们在每个层级利用了提前停止机制,并通过构建收敛谱序列证明了算法的快速收敛特性。所提出算法的性能和效率通过工程和量子系统中的实际示例得到了验证。
英文摘要:
Interest in reinforcement learning (RL) for large-scale systems, comprising extensive populations of intelligent agents interacting with heterogeneous environments, has surged significantly across diverse scientific domains in recent years. However, the large-scale nature of these systems often leads to high computational costs or reduced performance for most state-of-the-art RL techniques. To address these challenges, we propose a novel RL architecture and derive effective algorithms to learn optimal policies for arbitrarily large systems of agents. In our formulation, we model such systems as parameterized control systems defined on an infinite-dimensional function space. We then develop a moment kernel transform that maps the parameterized system and the value function into a reproducing kernel Hilbert space. This transformation generates a sequence of finite-dimensional moment representations for the RL problem, organized into a filtrated structure. Leveraging this RL filtration, we develop a hierarchical algorithm for learning optimal policies for the infinite-dimensional parameterized system. To enhance the algorithm's efficiency, we exploit early stopping at each hierarchy, demonstrating the fast convergence property of the algorithm through the construction of a convergent spectral sequence. The performance and efficiency of the proposed algorithm are validated using practical examples in engineering and quantum systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。