SwapMix:视觉问答中视觉语境过度依赖的诊断与规范
SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answering
摘要:
虽然视觉问答(VQA)进展迅速,以前的工作提出了目前的VQA模型的鲁棒性的关注。在这项工作中,我们从一个新的角度研究VQA模型的鲁棒性:视觉上下文。我们认为模型过度依赖于视觉环境,即,图像中不相关的物体来进行预测。为了诊断模型对视觉上下文的依赖并测量其鲁棒性,我们提出了一种简单而有效的扰动技术SwapMix。SwapMix通过将不相关上下文对象的特征与数据集中其他对象的特征交换来扰乱视觉上下文。使用SwapMix,我们能够为代表性VQA模型更改45%以上问题的答案。此外,我们训练的模型与完美的视觉,发现上下文过度依赖高度依赖于视觉表示的质量。除了诊断之外,SwapMix还可以在训练期间作为数据增强策略应用,以便规范上下文过度依赖。通过交换上下文对象特征,可以有效地抑制模型对上下文的依赖。使用SwapMix研究了两个代表性的VQA模型:共同注意模型MCAN和大规模预训练模型LXMERT。我们在流行的GQA数据集上的实验表明SwapMix在诊断模型鲁棒性和正则化过度依赖视觉上下文方面的有效性。我们方法的代码可在https://github.com/vipulgupta1011/swapmix上获得
英文摘要:
While Visual Question Answering (VQA) has progressed rapidly, previous works raise concerns about robustness of current VQA models. In this work, we study the robustness of VQA models from a novel perspective: visual context. We suggest that the models over-rely on the visual context, i.e., irrelevant objects in the image, to make predictions. To diagnose the model's reliance on visual context and measure their robustness, we propose a simple yet effective perturbation technique, SwapMix. SwapMix perturbs the visual context by swapping features of irrelevant context objects with features from other objects in the dataset. Using SwapMix we are able to change answers to more than 45 % of the questions for a representative VQA model. Additionally, we train the models with perfect sight and find that the context over-reliance highly depends on the quality of visual representations. In addition to diagnosing, SwapMix can also be applied as a data augmentation strategy during training in order to regularize the context over-reliance. By swapping the context object features, the model reliance on context can be suppressed effectively. Two representative VQA models are studied using SwapMix: a co-attention model MCAN and a large-scale pretrained model LXMERT. Our experiments on the popular GQA dataset show the effectiveness of SwapMix for both diagnosing model robustness and regularizing the over-reliance on visual context. The code for our method is available at https://github.com/vipulgupta1011/swapmix
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。