arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.06148cs.CVcs.AI
收藏

VLM-RobustBench: 一种全面的视觉语言模型鲁棒性基准

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

University of Edinburgh, UK(爱丁堡大学)

Rohit Saxena, Alessandro Suglia, Pasquale Minervini

AI总结:

VLM-RobustBench评估了视觉语言模型在不同图像失真下的鲁棒性,发现低严重性空间扰动对性能影响更大,提示需改进鲁棒性评估和训练方法。

摘要:

视觉语言模型(VLMs)在标准高质量数据集上表现出色,但我们在真实世界图像失真下的表现仍不完全了解。我们提出了VLM-RobustBench,该基准覆盖49种增强类型,包括噪声、模糊、天气、数字和几何扰动,在不同严重程度(低/中/高)和二进制转换下进行评估,产生133种损坏设置。我们评估了四个家族(Qwen、InternVL、Molmo、Gemma)的VLMs在两个互补基准:MMBenchmark(视觉基础)和MMMU-Pro(推理导向)上的表现。我们的结果表明,视觉严重性是困难的弱预测因子:低严重性空间扰动往往比视觉严重光度损坏更破坏性能。特别是,低严重性玻璃模糊将MMBenchmark准确性平均降低约8个百分点,而最大的下降来自重采样和几何扰动(如upsample、elastic_transform),达到最高34个百分点。总体而言,我们的发现表明当前VLMs在语义上强大但空间上脆弱,这促使定义新的鲁棒性评估协议和训练制度,强调重采样和几何不变性。

英文摘要:

Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate VLMs from four families (Qwen, InternVL, Molmo, Gemma) on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass_blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic_transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑