arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.00828cs.CV

RoboStressBench: 在具身场景中基准测试VLM对物理视觉压力的鲁棒性

RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenes

HKUST(GZ)(香港科技大学(广州))

Leyi Wu, Yifan Zhao, Jinjie Zhang, Suzeyu Chen, Wosong Chen, Zhifei Chen, Tianshuo Xu, Qingchun He, Hongxin Hu, Haojian Huang, Yangkai Wei, Wenqian Li, Yinchuan Li, Ying-Cong Chen

AI总结:

本文提出RoboStressBench,从逆图形学角度将视觉压力分解为材质、视角、光照和几何四个物理维度,系统评估VLM在真实物理压力下的鲁棒性,并引入压力感知求解器提升高压力场景下的性能。

摘要:

视觉语言模型(VLM)展现出强大的视觉理解能力,并越来越多地部署在具身AI系统中,这些系统需要在真实条件下进行可靠的感知。然而,现有的基准测试使用干净图像或孤立扰动来评估VLM,而非由物理场景形成引起的压力。这种设计有两个局限性:它仅覆盖了日常视觉压力的一小部分子集,并且某些扰动在现实具身场景中很少出现。这一差距引发了一个基本问题:我们如何以一种原则性的方式定义视觉压力,以捕捉物理环境中遇到的各种因素?为了解决这个问题,我们从逆图形学角度构建视觉感知,并引入RoboStressBench,这是一个用于评估VLM在具身场景中对物理视觉压力鲁棒性的基准测试。受物理渲染方程的启发,RoboStressBench将视觉压力分解为四个物理基础维度:材质(M)、视角(V)、光照(L)和几何(G)。这种设计使RoboStressBench能够覆盖现实世界环境中的广泛视觉压力,同时允许对其在VLM能力(如视觉识别、推理和规划)上的影响进行受控分析。通过对最先进的VLM进行全面评估,我们识别出特定于压力的失败模式,并揭示了不同的物理因素会降低不同的具身能力,而这些往往被总体准确率所掩盖。我们进一步引入了一种压力感知的智能求解器,它在推理前检测视觉压力源并调用视觉编辑技能,从而提高了高压力场景下的鲁棒性。总体而言,RoboStressBench提供了一个原则性的评估框架,用于诊断和改进VLM在真实物理压力下的感知能力,支持开发更可靠的具身AI系统。

英文摘要:

Vision-Language Models (VLMs) have shown strong visual understanding and are increasingly deployed in embodied AI systems, where reliable perception under real conditions is essential. However, existing benchmarks assess VLMs using clean images or isolated perturbations rather than stresses caused by physical scene formation. This design has two limitations: it covers only a narrow subset of everyday visual stresses, and some perturbations rarely appear in realistic embodied scenes. This gap raises a fundamental question: how can we define visual stress in a principled way that captures the diverse factors encountered in physical environments? To address this question, we formulate visual perception from an inverse graphics perspective and introduce RoboStressBench, a benchmark for evaluating VLM robustness to physical visual stress in embodied scenes. Inspired by the physical rendering equation, RoboStressBench decomposes visual stress into four physically grounded dimensions: Material (M), Viewpoint (V), Lighting (L), and Geometry (G). This design enables RoboStressBench to cover a broad range of visual stresses in real-world environments, while allowing controlled analysis of their effects on VLM capabilities such as visual recognition, reasoning, and planning. Through comprehensive evaluations of state-of-the-art VLMs, we identify stress-specific failure modes and reveal that different physical factors degrade different embodied capabilities, which are often obscured by aggregate accuracy. We further introduce a stress-aware agentic solver that detects visual stressors and invokes visual-editing skills before reasoning, improving robustness in high-stress scenarios. Overall, RoboStressBench provides a principled evaluation framework for diagnosing and improving VLM perception under real-world physical stress, supporting the development of more reliable embodied AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑