用于终身强化学习中可扩展且鲁棒规划的最小价值等价部分模型
Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learning
AI总结：
针对终身强化学习中全环境模型难以实现可扩展鲁棒规划的问题，提出最小价值等价部分模型，给出形式化定义与理论优势，结合深度学习启发式学习方法，通过实验验证其在可扩展性和抗分布偏移、累积误差方面的有效性。
摘要：
从纯交互中学习环境模型通常被认为是构建终身强化学习智能体的核心组件。然而，基于模型的强化学习中的常见做法是学习对智能体环境各个方面都进行建模的模型，无论这些方面对于得出最优决策是否重要。在本文中，我们认为这类模型并不特别适用于在终身强化学习场景中执行可扩展且鲁棒的规划，因此我们提出了仅对环境相关方面进行建模的新型模型，我们称之为“最小价值等价部分模型（minimal value-equivalent partial models）”。在给出这些模型的形式化定义后，我们提供了理论结果，证明了使用此类模型执行规划在可扩展性方面的优势，随后开展实验以实证验证我们的理论结果。接着，我们就如何使用深度学习架构学习这类模型提供了一些实用的启发式方法，并通过实证证明，以这种方式学习的模型能够实现对分布偏移和累积模型误差具有鲁棒性的规划。总体而言，我们的理论和实证结果均表明，最小价值等价部分模型能够为终身强化学习场景中的可扩展鲁棒规划带来显著收益。
英文摘要：
Learning models of the environment from pure interaction is often considered an essential component of building lifelong reinforcement learning agents. However, the common practice in model-based reinforcement learning is to learn models that model every aspect of the agent's environment, regardless of whether they are important in coming up with optimal decisions or not. In this paper, we argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios and we propose new kinds of models that only model the relevant aspects of the environment, which we call "minimal value-equivalent partial models". After providing a formal definition for these models, we provide theoretical results demonstrating the scalability advantages of performing planning with such models and then perform experiments to empirically illustrate our theoretical results. Then, we provide some useful heuristics on how to learn these kinds of models with deep learning architectures and empirically demonstrate that models learned in such a way can allow for performing planning that is robust to distribution shifts and compounding model errors. Overall, both our theoretical and empirical results suggest that minimal value-equivalent partial models can provide significant benefits to performing scalable and robust planning in lifelong reinforcement learning scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
