VQAttack：基于预训练模型的视觉问答可迁移对抗攻击
VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models
AI总结：
本文提出VQAttack模型，利用预训练多模态源模型迭代生成图像和文本对抗扰动，通过LLM增强图像攻击与跨模态联合攻击模块，实现对目标VQA模型的高效可迁移攻击，揭示了预训练与微调范式的鲁棒性盲点。
摘要：
视觉问答（VQA）是计算机视觉和自然语言处理领域的一项基础任务。尽管“预训练与微调”学习范式显著提升了VQA性能，但该学习范式的对抗鲁棒性尚未得到探索。在本文中，我们深入探讨了一个新问题：利用预训练的多模态源模型创建对抗性图文对，随后将其迁移以攻击目标VQA模型。相应地，我们提出了一种新颖的VQAttack模型，该模型能够通过所设计的模块迭代生成图像和文本扰动：大语言模型（LLM）增强的图像攻击模块和跨模态联合攻击模块。在每次迭代中，LLM增强的图像攻击模块首先优化基于潜在表示的损失，以生成特征级图像扰动。随后，它结合LLM，通过优化所设计的掩码答案反恢复损失来进一步增强图像扰动。跨模态联合攻击模块将在特定迭代中被触发，按顺序更新图像和文本扰动。值得注意的是，文本扰动更新同时基于词嵌入空间中学习到的梯度和基于词同义词的替换。在两个VQA数据集和五个验证模型上的实验结果表明，与最先进的基线方法相比，所提出的VQAttack在可迁移攻击设置下具有有效性。这项工作揭示了VQA任务中“预训练与微调”范式的一个显著盲点。源代码将会发布。
英文摘要：
Visual Question Answering (VQA) is a fundamental task in computer vision and natural language process fields. Although the ``pre-training & finetuning'' learning paradigm significantly improves the VQA performance, the adversarial robustness of such a learning paradigm has not been explored. In this paper, we delve into a new problem: using a pre-trained multimodal source model to create adversarial image-text pairs and then transferring them to attack the target VQA models. Correspondingly, we propose a novel VQAttack model, which can iteratively generate both image and text perturbations with the designed modules: the large language model (LLM)-enhanced image attack and the cross-modal joint attack module. At each iteration, the LLM-enhanced image attack module first optimizes the latent representation-based loss to generate feature-level image perturbations. Then it incorporates an LLM to further enhance the image perturbations by optimizing the designed masked answer anti-recovery loss. The cross-modal joint attack module will be triggered at a specific iteration, which updates the image and text perturbations sequentially. Notably, the text perturbation updates are based on both the learned gradients in the word embedding space and word synonym-based substitution. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQAttack in the transferable attack setting, compared with state-of-the-art baselines. This work reveals a significant blind spot in the ``pre-training & fine-tuning'' paradigm on VQA tasks. Source codes will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
