arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10667cs.LG
收藏

UniZero:基于可扩展潜在世界模型的通用高效规划

UniZero: Generalized and Efficient Planning with Scalable Latent World Models

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)

Yuan Pu, Yazhe Niu, Zhenjie Yang, Jiyuan Ren, Hongsheng Li, Yu Liu

AI总结:

UniZero 提出基于模块化 transformer 的可扩展潜在世界模型,通过联合优化长时程世界模型与策略,在 Atari、DMControl 及多任务和长期记忆基准上实现高效规划并超越现有方法。

摘要:

学习预测性世界模型对于增强强化学习(RL)智能体的规划能力至关重要。近年来,MuZero 风格的算法利用价值等价原理和蒙特卡洛树搜索(MCTS),已在多个领域取得了超越人类水平的表现。然而,这些方法在具有多样依赖关系和任务变异性的异构场景中难以扩展。为克服这些局限,我们提出了 UniZero,这是一种新颖的方法,采用基于模块化 transformer 的世界模型来有效学习共享潜在空间。通过以学习到的潜在历史为条件,同时预测潜在动态和面向决策的量,UniZero 实现了长时程世界模型与策略的联合优化,从而促进在潜在空间中进行更广泛、更高效的规划。我们表明,UniZero 在需要长期记忆的基准测试中显著优于现有基线方法。此外,UniZero 在 Atari 基准上进行的多任务学习实验中展示了卓越的可扩展性。在标准的单任务 RL 设置(如 Atari 和 DMControl)中,UniZero 达到甚至超越了当前最先进方法的性能。最后,大量的消融研究和可视化分析验证了 UniZero 设计选择的有效性和可扩展性。我们的代码可在 https://github.com/opendilab/LightZero 获取。

英文摘要:

Learning predictive world models is crucial for enhancing the planning capabilities of reinforcement learning (RL) agents. Recently, MuZero-style algorithms, leveraging the value equivalence principle and Monte Carlo Tree Search (MCTS), have achieved superhuman performance in various domains. However, these methods struggle to scale in heterogeneous scenarios with diverse dependencies and task variability. To overcome these limitations, we introduce UniZero, a novel approach that employs a modular transformer-based world model to effectively learn a shared latent space. By concurrently predicting latent dynamics and decision-oriented quantities conditioned on the learned latent history, UniZero enables joint optimization of the long-horizon world model and policy, facilitating broader and more efficient planning in the latent space. We show that UniZero significantly outperforms existing baselines in benchmarks that require long-term memory. Additionally, UniZero demonstrates superior scalability in multitask learning experiments conducted on Atari benchmarks. In standard single-task RL settings, such as Atari and DMControl, UniZero matches or even surpasses the performance of current state-of-the-art methods. Finally, extensive ablation studies and visual analyses validate the effectiveness and scalability of UniZero's design choices. Our code is available at \textcolor{magenta}{https://github.com/opendilab/LightZero}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑