视觉-语言模型的协同鲁棒性评估框架
Coordinated Robustness Evaluation Framework for Vision-Language Models
AI总结：
本文提出协同对抗攻击框架，通过通用代理模型生成图像与文本联合扰动，以评估视觉-语言模型的跨模态鲁棒性，并在 VQA 等任务上验证其优于现有攻击。
摘要：
视觉-语言模型整合了计算机视觉与自然语言处理能力，已在图像描述和视觉问答等任务中展现出显著进展。然而，与传统模型类似，它们容易受到微小扰动的影响，这给其鲁棒性带来了挑战，尤其是在部署场景中。评估这些模型的鲁棒性需要同时在视觉和语言两种模态上施加扰动，以了解其模态间依赖关系。在本工作中，我们训练了一个通用代理模型，该模型能够同时接收图像和文本作为输入，并生成联合表示，该表示进一步用于生成文本和图像两种模态的对抗扰动。我们使用多种最先进的视觉-语言模型，在视觉问答和视觉推理数据集上对这种协同攻击策略进行了评估。结果表明，所提出的策略优于近期文献中的其他多模态攻击和单模态攻击。我们的结果证明了该策略在破坏若干最先进预训练多模态模型（例如 instruct-BLIP、ViLT 等）鲁棒性方面的有效性。
英文摘要：
Vision-language models, which integrate computer vision and natural language processing capabilities, have demonstrated significant advancements in tasks such as image captioning and visual question and answering. However, similar to traditional models, they are susceptible to small perturbations, posing a challenge to their robustness, particularly in deployment scenarios. Evaluating the robustness of these models requires perturbations in both the vision and language modalities to learn their inter-modal dependencies. In this work, we train a generic surrogate model that can take both image and text as input and generate joint representation which is further used to generate adversarial perturbations for both the text and image modalities. This coordinated attack strategy is evaluated on the visual question and answering and visual reasoning datasets using various state-of-the-art vision-language models. Our results indicate that the proposed strategy outperforms other multi-modal attacks and single-modality attacks from the recent literature. Our results demonstrate their effectiveness in compromising the robustness of several state-of-the-art pre-trained multi-modal models such as instruct-BLIP, ViLT and others.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
