arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.12985cs.AI
收藏

SENTINEL:面向基础模型基于具身代理的安全性评估多级形式框架

SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents

Northwestern University, USA;University of Southern California, USA;College of William \& Mary
Simon Sinong Zhan, Philip Wang, Yao Liu, Yiyan Peng, Zinan Wang, Qineng Wang, Zhian Ruan, Xiangyu Shi, Xinyu Cao, Frank Yang, Zhenyang Ni, Kangrui Wang, Ruohan … 展开作者

Simon Sinong Zhan, Philip Wang, Yao Liu, Yiyan Peng, Zinan Wang, Qineng Wang, Zhian Ruan, Xiangyu Shi, Xinyu Cao, Frank Yang, Zhenyang Ni, Kangrui Wang, Ruohan Zhang, Huajie Shao, Manling Li, Qi Zhu

AI总结:

SENTINEL通过形式时序逻辑多级验证框架,系统性评估基础模型具身代理在模拟环境中的物理安全性。

摘要:

我们提出了SENTINEL,一种用于正式评估基础模型(FM)基于具身代理的物理安全性的框架。SENTINEL是首个在统一形式框架内提供多级安全性评估的系统,涵盖语义解释、计划生成和物理执行。不同于以往依赖启发式规则或主观FM判断的方法,SENTINEL将实际安全需求根植于形式时序逻辑(TL)语义中,能够精确指定状态不变量、时序依赖性和时间约束。它采用多级验证流程,其中(i)在语义层面,直观的自然语言安全需求被形式化为TL公式,代理对这些要求的理解被探测以与TL公式对齐;(ii)在计划层面,由代理生成的高层行动计划和子目标被验证以TL公式以检测不安全计划;(iii)在轨迹层面,多个执行轨迹被合并为计算树,并高效验证以物理细节的TL规范进行最终安全性检查。我们将在VirtualHome和AI2-THOR中应用SENTINEL,并对多个基于FM的具身代理正式评估多种安全需求。我们的实验表明,通过将物理安全根植于时序逻辑并应用多级验证方法,SENTINEL为在基于模拟的物理环境中系统性评估FM基于具身代理的安全性提供了严谨的基础,并能有效暴露在解释、计划和执行任务中的潜在安全违规。

英文摘要:

We present SENTINEL, a framework for formally evaluating the physical safety of foundation model (FM)-based embodied agents. SENTINEL is the first to provide multi-level safety evaluation across semantic interpretation, plan generation, and physical execution within a unified formal framework. Unlike prior methods that rely on heuristic rules or subjective FM judgments, SENTINEL grounds practical safety requirements in formal temporal logic (TL) semantics that can precisely specify state invariants, temporal dependencies, and timing constraints. It employs a multi-level verification pipeline where (i) at the semantic level, intuitive natural language safety requirements are formalized into TL formulas and the agent's understanding of these requirements is probed for alignment with the TL formulas; (ii) at the plan level, high-level action plans and subgoals generated by the agent are verified against the TL formulas to detect unsafe plans before execution; and (iii) at the trajectory level, multiple execution trajectories are merged into a computation tree and efficiently verified against physically-detailed TL specifications for a final safety check. We apply SENTINEL in VirtualHome and AI2-THOR, and formally evaluate multiple FM-based embodied agents against diverse safety requirements. Our experiments show that by grounding physical safety in temporal logic and applying verification methods across multiple levels, SENTINEL provides a rigorous foundation for systematically evaluating the safety of FM-based embodied agents in simulation-based physical environments, and can effectively expose potential safety violations in interpreting, planning, and executing the tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑