发表机构
Massachusetts Institute of Technology（麻省理工学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出一种安全元强化学习框架，通过在信息空间中推理安全性并学习安全值函数，实现适应过程中的安全过滤与约束策略优化，实验验证了其有效性。
AI 中文摘要
元强化学习（meta-RL）使智能体能够通过有限的经验适应未见过的任务。尽管其前景广阔，但元强化学习在现实世界任务中的应用受到安全要求的阻碍，而先前的工作对此探索不足。在本文中，我们提出了一个在适应过程中明确考虑安全性的安全元强化学习框架。我们的关键见解是在信息空间中进行安全性推理，该空间同时捕捉物理状态和智能体对底层任务的信念。在此空间内，我们引入了一个安全值函数，用于衡量智能体无限期避免进入不安全区域的概率。我们证明该函数满足自一致性条件和贝尔曼方程，使其可以通过元强化学习进行学习。基于这一公式，我们开发了一种安全元强化学习算法，该算法学习安全值函数，并将其用于安全过滤和约束策略优化。在元强化学习基准上的实验证明了所提出方法的有效性。
英文摘要
Meta-reinforcement learning (meta-RL) enables agents to adapt to unseen tasks with limited experience. Despite its promise, the application of meta-RL in real-world tasks is hindered by safety requirements, which have been underexplored in prior work. In this paper, we propose a safe meta-RL framework that explicitly accounts for safety during adaptation. Our key insight is to reason about safety in the information space, which captures both the physical state and the agent's belief over the underlying task. Within this space, we introduce a safety value function that measures the probability of the agent avoiding unsafe regions indefinitely. We show that this function satisfies a self-consistency condition and a Bellman equation, which make it learnable via meta-RL. Based on this formulation, we develop a safe meta-RL algorithm that learns the safety value function and leverages it for safety filtering and constrained policy optimization. Experiments on meta-RL benchmarks demonstrate the effectiveness of the proposed method.
