arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.10567cs.CV
收藏

CL-CrossVQA:跨领域视觉问答的持续学习基准

CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering

University of Oxford(牛津大学);Siemens AG(西门子股份公司);LMU Munich(慕尼黑大学);Technical University of Munich(慕尼黑工业大学)

Yao Zhang, Haokun Chen, Ahmed Frikha, Yezi Yang, Denis Krompass, Gengyuan Zhang, Jindong Gu, Volker Tresp

摘要:

视觉问答是一个多学科交叉的研究课题。为了产生正确的答案,它需要理解图像的视觉内容、自然语言问题,以及对图像中包含的信息和世界知识的常识推理。最近,大规模的视觉和语言预训练模型(VLPMs)由于其优越的性能已经成为VQA任务的主流方法。标准的做法是使用特定于域的VQA数据集对在大型通用域数据集上预先训练的大规模VLPM进行微调。然而,实际上,应用领域可能会随着时间的推移而变化,这就要求VLPM不断学习和适应新的领域,而不会忘记以前获得的知识。现有的持续学习研究大多集中在单峰任务上,而更实际的应用场景,即跨领域VQA的持续学习还没有被研究。基于此,我们引入了CL-CrossVQA,一个严格的跨领域视觉问答的持续学习基准,通过它,我们对来自不同领域的4种VLPM,4种CL方法和5个VQA数据集进行了广泛的实验。此外,通过探索中间层的遗忘现象,我们提供了深入了解模型架构如何影响CL性能,为什么CL方法可以在一定程度上帮助减轻VLPM中的遗忘,以及如何设计适合于VLPM的CL方法在这个具有挑战性的持续学习环境。为了促进CL跨域VQA的未来工作,我们将发布我们的数据集和代码。

英文摘要:

Visual Question Answering (VQA) is a multi-discipline research task. To produce the right answer, it requires an understanding of the visual content of images, the natural language questions, as well as commonsense reasoning over the information contained in the image and world knowledge. Recently, large-scale Vision-and-Language Pre-trained Models (VLPMs) have been the mainstream approach to VQA tasks due to their superior performance. The standard practice is to fine-tune large-scale VLPMs pre-trained on huge general-domain datasets using the domain-specific VQA datasets. However, in reality, the application domain can change over time, necessitating VLPMs to continually learn and adapt to new domains without forgetting previously acquired knowledge. Most existing continual learning (CL) research concentrates on unimodal tasks, whereas a more practical application scenario, i.e, CL on cross-domain VQA, has not been studied. Motivated by this, we introduce CL-CrossVQA, a rigorous Continual Learning benchmark for Cross-domain Visual Question Answering, through which we conduct extensive experiments on 4 VLPMs, 4 CL approaches, and 5 VQA datasets from different domains. In addition, by probing the forgetting phenomenon of the intermediate layers, we provide insights into how model architecture affects CL performance, why CL approaches can help mitigate forgetting in VLPMs to some extent, and how to design CL approaches suitable for VLPMs in this challenging continual learning environment. To facilitate future work on CL for cross-domain VQA, we will release our datasets and code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑