arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.06407cs.CVcs.AI
收藏

思考与观察的结合用于外部知识视觉问答

Combo of Thinking and Observing for Outside-Knowledge VQA

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);ByteDance AI Lab(字节跳动人工智能实验室)

Qingyi Si, Yuchen Mo, Zheng Lin, Huishan Ji, Weiping Wang

AI总结:

针对外部知识视觉问答,提出将跨模态空间与自然语言空间对齐的框架,融合多模态与文本知识,准确率超越最先进方法6.17%。

摘要:

外部知识视觉问答是一项具有挑战性的任务,既需要获取开放式的真实世界知识,也需要使用这些知识。现有的一些解决方案将外部知识引入跨模态空间,这忽视了自然语言空间中更为丰富的文本知识;而另一些解决方案则将图像转换为文本,进一步与文本知识融合到自然语言空间中,完全放弃了视觉特征的使用。在本文中,我们受到启发,将跨模态空间约束到与自然语言空间相同的空间中,这样视觉特征得以直接保留,同时模型仍能受益于自然语言空间中丰富的知识。为此,我们提出了一种新颖的框架,包含多模态编码器、文本编码器和答案解码器。这种结构使我们能够引入更多类型的知识,包括显式和隐式的多模态及文本知识。大量实验验证了所提方法的优越性,其准确率比现有最先进方法高出6.17%。我们还对每个组件进行了全面的消融研究,并系统地研究了不同类型知识的作用。代码和知识数据可在 https://github.com/PhoebusSi/Thinking-while-Observing 获取。

英文摘要:

Outside-knowledge visual question answering is a challenging task that requires both the acquisition and the use of open-ended real-world knowledge. Some existing solutions draw external knowledge into the cross-modality space which overlooks the much vaster textual knowledge in natural-language space, while others transform the image into a text that further fuses with the textual knowledge into the natural-language space and completely abandons the use of visual features. In this paper, we are inspired to constrain the cross-modality space into the same space of natural-language space which makes the visual features preserved directly, and the model still benefits from the vast knowledge in natural-language space. To this end, we propose a novel framework consisting of a multimodal encoder, a textual encoder and an answer decoder. Such structure allows us to introduce more types of knowledge including explicit and implicit multimodal and textual knowledge. Extensive experiments validate the superiority of the proposed method which outperforms the state-of-the-art by 6.17% accuracy. We also conduct comprehensive ablations of each component, and systematically study the roles of varying types of knowledge. Codes and knowledge data can be found at https://github.com/PhoebusSi/Thinking-while-Observing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑