AdvDreamer揭示:视觉-语言模型真的准备好应对真实世界3D变化了吗?
AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?
AI总结:
AdvDreamer提出首个从单视角生成物理可复现对抗性3D变换样本的框架,并构建MM3DTBench基准,系统揭示真实世界3D变化对视觉-语言模型鲁棒性的严重威胁。
摘要:
视觉-语言模型(VLMs)已展现出卓越的泛化能力,但它们在动态真实世界场景中的鲁棒性仍未得到充分探索。为了系统评估VLMs对真实世界3D变化的鲁棒性,我们提出了AdvDreamer,这是首个能够从单视角观测中生成物理可复现的对抗性3D变换(Adv-3DT)样本的框架。在AdvDreamer中,我们整合了三项关键创新:首先,为了在有限先验知识下精确刻画真实世界3D变化,我们设计了一个基于生成式3D先验的零样本单目姿态操纵流程。其次,为了确保最坏情况Adv-3DT样本的视觉质量,我们提出了一个自然度奖励模型,在对抗优化过程中提供连续的自然度正则化,有效防止收敛到幻觉或不自然元素。第三,为了支持跨不同VLM架构和视觉-语言任务的系统评估,我们引入了逆语义概率损失作为对抗优化目标,该目标仅在基础视觉-文本对齐空间中运作。基于捕获到的高攻击性和高迁移性的Adv-3DT样本,我们建立了MM3DTBench,这是首个专门用于评估VLM在具有挑战性的3D变化下鲁棒性的VQA基准数据集。对具有不同架构的代表性VLMs的广泛评估表明,真实世界3D变化可对各种任务中的模型性能构成严重威胁。
英文摘要:
Vision Language Models (VLMs) have exhibited remarkable generalization capabilities, yet their robustness in dynamic real-world scenarios remains largely unexplored. To systematically evaluate VLMs' robustness to real-world 3D variations, we propose AdvDreamer, the first framework capable of generating physically reproducible Adversarial 3D Transformation (Adv-3DT) samples from single-view observations. In AdvDreamer, we integrate three key innovations: Firstly, to characterize real-world 3D variations with limited prior knowledge precisely, we design a zero-shot Monocular Pose Manipulation pipeline built upon generative 3D priors. Secondly, to ensure the visual quality of worst-case Adv-3DT samples, we propose a Naturalness Reward Model that provides continuous naturalness regularization during adversarial optimization, effectively preventing convergence to hallucinated or unnatural elements. Thirdly, to enable systematic evaluation across diverse VLM architectures and visual-language tasks, we introduce the Inverse Semantic Probability loss as the adversarial optimization objective, which solely operates in the fundamental visual-textual alignment space. Based on the captured Adv-3DT samples with high aggressiveness and transferability, we establish MM3DTBench, the first VQA benchmark dataset tailored to evaluate VLM robustness under challenging 3D variations. Extensive evaluations of representative VLMs with varying architectures reveal that real-world 3D variations can pose severe threats to model performance across various tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。