分析视觉-语言模型对常见损坏的鲁棒性
Analysing the Robustness of Vision-Language-Models to Common Corruptions
AI总结：
本研究首次全面分析视觉-语言模型对19种常见图像损坏的鲁棒性，引入TextVQA-C和GQA-C基准，揭示Transformer模型在不同任务中的脆弱性模式及其频域解释，为开发更鲁棒的模型提供见解。
摘要：
视觉-语言模型（VLMs）在理解和推理视觉及文本内容方面展现了令人印象深刻的能力。然而，它们对常见图像损坏的鲁棒性仍未得到充分探索。在这项工作中，我们首次对VLM鲁棒性进行了全面分析，涵盖了来自ImageNet-C基准的19种损坏类型，分为四类：噪声、模糊、天气和数字失真。我们引入了两个新基准，TextVQA-C和GQA-C，以系统地评估损坏如何分别影响场景文本理解和基于对象的推理。我们的分析揭示，基于Transformer的VLM在不同任务中表现出不同的脆弱性模式：文本识别在模糊和雪损坏下退化最为严重，而对象推理对霜冻和脉冲噪声等损坏表现出更高的敏感性。我们将这些观察与不同损坏的频域特征联系起来，揭示了Transformer对低频处理的内在偏向如何解释其差异化的鲁棒性模式。我们的发现为开发更抗损坏的视觉-语言模型以应用于现实世界提供了宝贵的见解。
英文摘要：
Vision-language models (VLMs) have demonstrated impressive capabilities in understanding and reasoning about visual and textual content. However, their robustness to common image corruptions remains under-explored. In this work, we present the first comprehensive analysis of VLM robustness across 19 corruption types from the ImageNet-C benchmark, spanning four categories: noise, blur, weather, and digital distortions. We introduce two new benchmarks, TextVQA-C and GQA-C, to systematically evaluate how corruptions affect scene text understanding and object-based reasoning, respectively. Our analysis reveals that transformer-based VLMs exhibit distinct vulnerability patterns across tasks: text recognition deteriorates most severely under blur and snow corruptions, while object reasoning shows higher sensitivity to corruptions such as frost and impulse noise. We connect these observations to the frequency-domain characteristics of different corruptions, revealing how transformers' inherent bias toward low-frequency processing explains their differential robustness patterns. Our findings provide valuable insights for developing more corruption-robust vision-language models for real-world applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
