arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.04692cs.CV
收藏

语言优先不是唯一的障碍--VQA的一个可持续学习的基准

Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Tencent Inc(腾讯公司);Peking University(北京大学)

Qingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin, Yuanxin Liu, Peng Fu, Yanan Cao, Weiping Wang, Jie Zhou

摘要:

视觉问答(VQA)模型倾向于学习由数据集偏差形成的捷径解决方案,而不是预期的解决方案。为了评估VQA模型在快捷学习之外的推理能力,VQA-CP v2数据集在给定问题类型的训练集和测试集之间引入了分布偏移。通过这种方式,模型无法使用训练集快捷方式(从问题类型到答案)在测试集上表现良好。然而,VQA-CP v2只考虑一种类型的捷径,因此仍然不能保证模型依赖于预期的解决方案,而不是特定于此捷径的解决方案。为了克服这一限制,我们提出了一个新的数据集,认为不同类型的捷径,通过构建不同的分布偏移在多个OOD测试集。此外,我们克服了VQA-CP v2使用中的三个令人不安的做法,例如,使用面向对象的测试集进行模型选择,进一步规范面向对象的评估流程。我们的基准测试为VQA中的快捷学习提供了更严格和全面的测试平台。我们基准最近的方法,并发现专门为特定的捷径设计的方法不能同时推广到我们不同的OOD测试集。本文还系统地研究了不同的捷径,并提供了一些有价值的发现,这可能会促进VQA中捷径学习的探索。

英文摘要:

Visual Question Answering (VQA) models are prone to learn the shortcut solution formed by dataset biases rather than the intended solution. To evaluate the VQA models' reasoning ability beyond shortcut learning, the VQA-CP v2 dataset introduces a distribution shift between the training and test set given a question type. In this way, the model cannot use the training set shortcut (from question type to answer) to perform well on the test set. However, VQA-CP v2 only considers one type of shortcut and thus still cannot guarantee that the model relies on the intended solution rather than a solution specific to this shortcut. To overcome this limitation, we propose a new dataset that considers varying types of shortcuts by constructing different distribution shifts in multiple OOD test sets. In addition, we overcome the three troubling practices in the use of VQA-CP v2, e.g., selecting models using OOD test sets, and further standardize OOD evaluation procedure. Our benchmark provides a more rigorous and comprehensive testbed for shortcut learning in VQA. We benchmark recent methods and find that methods specifically designed for particular shortcuts fail to simultaneously generalize to our varying OOD test sets. We also systematically study the varying shortcuts and provide several valuable findings, which may promote the exploration of shortcut learning in VQA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑