arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.15002cs.CLcs.CV
收藏

CommVQA:将视觉问答置于交际语境中

CommVQA: Situating Visual Question Answering in Communicative Contexts

Stanford University(斯坦福大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Nandita Shankar Naik, Christopher Potts, Elisa Kreiss

AI总结:

针对现有VQA模型脱离语境训练评估的问题,提出含1000张图、8949个问答对的交际语境VQA数据集CommVQA,实验发现当前模型存在幻觉率高等问题,证明上下文信息对提升VQA性能至关重要。

摘要:

当前的视觉问答(VQA)模型通常是在孤立的图像-问题对上进行训练和评估的。然而,人们提出的问题取决于他们的信息需求以及对图像内容的先验知识。为了评估将图像置于自然语境中如何塑造视觉问题,我们提出了CommVQA,这是一个VQA数据集,包含图像、图像描述、图像可能出现的真实交际场景(例如旅游网站),以及基于场景和描述的后续问答对。CommVQA包含1000张图像和8949个问答对,对当前模型构成了挑战。错误分析和一项人类受试者研究表明,生成的答案仍然存在很高的幻觉率,无法恰当地处理无法回答的问题,也不能适当地反映上下文信息。总体而言,我们表明获取上下文信息对于解决CommVQA任务至关重要,能带来性能最高的VQA模型,并凸显了将系统置于交际场景中的重要性。

英文摘要:

Current visual question answering (VQA) models tend to be trained and evaluated on image-question pairs in isolation. However, the questions people ask are dependent on their informational needs and prior knowledge about the image content. To evaluate how situating images within naturalistic contexts shapes visual questions, we introduce CommVQA, a VQA dataset consisting of images, image descriptions, real-world communicative scenarios where the image might appear (e.g., a travel website), and follow-up questions and answers conditioned on the scenario and description. CommVQA, which contains 1000 images and 8,949 question-answer pairs, poses a challenge for current models. Error analyses and a human-subjects study suggest that generated answers still contain high rates of hallucinations, fail to fittingly address unanswerable questions, and don't suitably reflect contextual information. Overall, we show that access to contextual information is essential for solving CommVQA, leading to the highest performing VQA model and highlighting the relevance of situating systems within communicative scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑