重新评估视觉问答中的评估实践：关于分布外泛化的案例研究
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
AI总结：
本文通过跨数据集评估发现，预训练视觉与语言模型在VQA分布外设置下泛化能力差，倾向于解决基准而非学习高级技能，且自动评估指标的严苛性会错误惩罚正确响应。
摘要：
在大规模多模态数据上预训练的视觉与语言（V&L）模型在图像描述和视觉问答（VQA）等任务上表现出色。此类模型的质量通常通过在未见数据上测量其性能来评估，这些数据通常与训练数据来自同一分布。然而，在VQA的分布外（数据集外）设置下进行评估时，我们观察到这些模型表现出较差的泛化能力。我们通过进行跨数据集评估，全面评估了两种预训练的V&L模型在不同设置（即分类和开放式文本生成）下的表现。我们发现这些模型倾向于学习解决基准测试，而不是学习VQA任务所需的高级技能。我们还发现，在大多数情况下，与判别式模型相比，生成式模型对数据分布的偏移不太敏感，并且多模态预训练通常有助于OOD泛化。最后，我们重新审视了自动VQA评估指标使用背后的假设，并实证表明其严苛性质会反复惩罚给出正确响应的模型。
英文摘要：
Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA). The quality of such models is commonly assessed by measuring their performance on unseen data that typically comes from the same distribution as the training data. However, when evaluated under out-of-distribution (out-of-dataset) settings for VQA, we observe that these models exhibit poor generalization. We comprehensively evaluate two pretrained V&L models under different settings (i.e. classification and open-ended text generation) by conducting cross-dataset evaluations. We find that these models tend to learn to solve the benchmark, rather than learning the high-level skills required by the VQA task. We also find that in most cases generative models are less susceptible to shifts in data distribution compared to discriminative ones, and that multimodal pretraining is generally helpful for OOD generalization. Finally, we revisit assumptions underlying the use of automatic VQA evaluation metrics, and empirically show that their stringent nature repeatedly penalizes models for correct responses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
