受大脑启发的规划以提升强化学习的泛化能力
Brain-Inspired Planning for Better Generalization in Reinforcement Learning
AI总结：
本论文受人类大脑规划机制启发，提出空间抽象注意力、Skipper任务分解框架和可行性评估器，以提升强化学习智能体的零样本系统化泛化能力并避免妄想性规划。
摘要：
现有的强化学习（RL）系统在应用于现实场景时面临重大挑战，主要原因是它们在不同于训练条件的环境中的泛化能力较差。本论文探索了通过赋予RL智能体推理行为来增强其零样本系统化泛化能力的方向，这些推理行为已被发现有助于人脑中的系统化泛化。受人类有意识规划行为的启发，我们首先引入了一种自上而下的注意力机制，该机制允许决策时规划智能体根据其即时意图，动态地将推理聚焦于环境状态中最相关的方面，我们将这一过程称为“空间抽象”。该方法显著提升了训练任务之外的系统化泛化能力。随后，在空间抽象的基础上，我们开发了Skipper框架，用于自动将复杂任务分解为更简单、更易管理的子任务。Skipper通过关注环境中相关的空间和时间元素，对分布偏移具有鲁棒性，并在长期组合规划中表现出有效性。最后，我们识别了依赖生成模型在规划过程中生成状态目标的规划智能体中常见的一种失效模式和安全风险。研究表明，大多数智能体会盲目信任它们幻觉生成的目标，从而导致妄想性规划行为。受人类大脑如何拒绝妄想性意图的启发，我们提出学习一个可行性评估器，以使其能够拒绝幻觉生成的不可行目标，这在多种规划智能体中带来了显著的性能提升。最后，我们提出了未来研究方向的建议，旨在实现通用的任务抽象并充分实现抽象规划。
英文摘要：
Existing Reinforcement Learning (RL) systems encounter significant challenges when applied to real-world scenarios, primarily due to poor generalization across environments that differ from their training conditions. This thesis explores the direction of enhancing agents' zero-shot systematic generalization abilities by granting RL agents reasoning behaviors that are found to help systematic generalization in the human brain. Inspired by human conscious planning behaviors, we first introduced a top-down attention mechanism, which allows a decision-time planning agent to dynamically focus its reasoning on the most relevant aspects of the environmental state given its instantaneous intentions, a process we call "spatial abstraction". This approach significantly improves systematic generalization outside the training tasks. Subsequently, building on spatial abstraction, we developed the Skipper framework to automatically decompose complex tasks into simpler, more manageable sub-tasks. Skipper provides robustness against distributional shifts and efficacy in long-term, compositional planning by focusing on pertinent spatial and temporal elements of the environment. Finally, we identified a common failure mode and safety risk in planning agents that rely on generative models to generate state targets during planning. It is revealed that most agents blindly trust the targets they hallucinate, resulting in delusional planning behaviors. Inspired by how the human brain rejects delusional intentions, we propose learning a feasibility evaluator to enable rejecting hallucinated infeasible targets, which led to significant performance improvements in various kinds of planning agents. Finally, we suggest directions for future research, aimed at achieving general task abstraction and fully enabling abstract planning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
