arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2308.00295cs.CVcs.AIcs.CL
收藏

让 Text-VQA 中的 V 变得重要

Making the V in Text-VQA Matter

KLE Technological University(KLE科技大学);IIIT Hyderabad(印度海得拉巴国际信息技术学院)

Shamanthak Hegde, Soumya Jahagirdar, Shankar Gangisetty

AI总结:

针对TextVQA数据集偏重文本导致模型忽略视觉特征的问题,提出结合TextVQA和VQA数据集训练的方法,增强图像与文本特征的相关性,提升问答性能。

摘要:

基于文本的 VQA 旨在通过读取图像中出现的文本来回答问题。与 VQA 任务相比,它需要大量的场景文本关系理解。最近的研究表明,数据集中的问答对更关注图像中出现的文本,而对视觉特征的重视较少,且一些问题不需要理解图像。在该数据集上训练的模型由于缺乏对视觉上下文的理解,会预测出有偏见的答案。例如,在“指示牌上写了什么?”这样的问题中,模型预测的答案总是“STOP”,这使得模型忽略了图像。为了解决这些问题,我们提出了一种方法,使用 VQA 数据集作为基于文本的 VQA 的外部知识,学习视觉特征(使 TextVQA 中的 V 变得重要)以及 OCR 特征和问题特征。具体来说,我们将 TextVQA 数据集和 VQA 数据集结合起来,在这个组合数据集上训练模型。这种简单而有效的方法增加了图像特征与图像中文本之间的理解和相关性,有助于更好地回答问题。我们进一步在不同数据集上测试了模型,并比较了它们的定性和定量结果。

英文摘要:

Text-based VQA aims at answering questions by reading the text present in the images. It requires a large amount of scene-text relationship understanding compared to the VQA task. Recent studies have shown that the question-answer pairs in the dataset are more focused on the text present in the image but less importance is given to visual features and some questions do not require understanding the image. The models trained on this dataset predict biased answers due to the lack of understanding of visual context. For example, in questions like "What is written on the signboard?", the answer predicted by the model is always "STOP" which makes the model to ignore the image. To address these issues, we propose a method to learn visual features (making V matter in TextVQA) along with the OCR features and question features using VQA dataset as external knowledge for Text-based VQA. Specifically, we combine the TextVQA dataset and VQA dataset and train the model on this combined dataset. Such a simple, yet effective approach increases the understanding and correlation between the image features and text present in the image, which helps in the better answering of questions. We further test the model on different datasets and compare their qualitative and quantitative results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑