arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09205cs.LG
收藏

用于训练鲁棒世界模型的无奖励课程

Reward-Free Curricula for Training Robust World Models

University of Oxford(牛津大学);University College London(伦敦大学学院)

Marc Rigter, Minqi Jiang, Ingmar Posner

AI总结:

本文研究无奖励探索中训练鲁棒世界模型的课程生成问题,提出依据各环境模型估计误差选择数据收集环境的 WAKER 算法,并实验证明其提升鲁棒性、效率与泛化。

摘要:

近来,人们对开发具备通用能力、能够无需在环境中额外训练即可适应新任务的智能体产生了浓厚兴趣。从无奖励探索中学习世界模型是一种很有前景的方法,它使得策略能够利用针对新任务的想象经验进行训练。然而,实现通用智能体需要在不同环境之间具备鲁棒性。在本文中,我们解决了一个新问题:在无奖励设定下生成课程,以训练鲁棒的世界模型。我们将鲁棒性定义为在所有环境实例上的极小极大遗憾,并表明极小极大遗憾可以与最小化世界模型在各环境实例中的最大误差联系起来。这一结果启发了我们的算法 WAKER:Weighted Acquisition of Knowledge across Environments for Robustness(为鲁棒性跨环境加权获取知识)。WAKER 根据每个环境下世界模型的估计误差选择用于数据收集的环境。我们的实验表明,WAKER 优于若干基线方法,带来了更好的鲁棒性、效率和泛化能力。

英文摘要:

There has been a recent surge of interest in developing generally-capable agents that can adapt to new tasks without additional training in the environment. Learning world models from reward-free exploration is a promising approach, and enables policies to be trained using imagined experience for new tasks. However, achieving a general agent requires robustness across different environments. In this work, we address the novel problem of generating curricula in the reward-free setting to train robust world models. We consider robustness in terms of minimax regret over all environment instantiations and show that the minimax regret can be connected to minimising the maximum error in the world model across environment instances. This result informs our algorithm, WAKER: Weighted Acquisition of Knowledge across Environments for Robustness. WAKER selects environments for data collection based on the estimated error of the world model for each environment. Our experiments demonstrate that WAKER outperforms several baselines, resulting in improved robustness, efficiency, and generalisation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑