视觉-语言-动作机器人元测试
Metamorphic Testing of Vision-Language Action-Enabled Robots
AI总结:
本文探讨元测试能否缓解视觉-语言-动作机器人测试 oracle 问题,提出两种关系模式和五种关系,通过实验证明其有效性,并展示其在不同任务和机器人上的通用性。
摘要:
视觉-语言-动作(VLA)模型是多模态机器人任务控制器,根据指令和视觉输入生成一系列低层控制动作,使机器人在物理环境中执行请求任务。这些系统面临测试 oracle 问题,传统方法定义复杂且不通用,而现有 oracle 通常仅评估符号表示,无法评估 VLA 机器人执行任务的质量。本文探讨元测试(MT)能否缓解此问题,提出两种关系模式和五种关系,通过五种 VLA 模型、两个模拟机器人和四个任务的实验证实 MT 能自动检测多种故障,包括未完成任务。更重要的是,所提 MRs 具有通用性,适用于不同 VLA 模型、机器人和任务,即使没有测试 oracle 也能应用。
英文摘要:
Vision-Language-Action (VLA) models are multimodal robotic task controllers that, given an instruction and visual inputs, produce a sequence of low-level control actions (or motor commands) enabling a robot to execute the requested task in the physical environment. These systems face the test oracle problem from multiple perspectives. On the one hand, a test oracle must be defined for each instruction prompt, which is a complex and non-generalizable approach. On the other hand, current state-of-the-art oracles typically capture symbolic representations of the world (e.g., robot and object states), enabling the correctness evaluation of a task, but fail to assess other critical aspects, such as the quality with which VLA-enabled robots perform a task. In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。