arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.06035cs.CL
收藏

语言和视觉推理的跨模态关联

Cross-Modality Relevance for Reasoning on Language and Vision

Michigan State University(密歇根州立大学)

Chen Zheng, Quan Guo, Parisa Kordjamshidi

摘要:

这项工作涉及的挑战,学习和推理的语言和视觉数据的相关下游任务,如视觉问答(VQA)和自然语言的视觉推理(NLVR)。我们设计了一种新的跨模态相关性模块,用于在端到端的框架中学习目标任务的监督下,各种输入模态的组件之间的相关性表示,这是更一般化的未观察到的数据相比,只是重塑原始的表示空间。除了对文本实体和视觉实体之间的相关性进行建模之外,我们还对文本中的实体关系和图像中的对象关系之间的高阶相关性进行建模。我们提出的方法在使用公共基准的两种不同的语言和视觉任务上表现出有竞争力的性能,并改进了最先进的已发表结果。通过NLVR任务学习的输入空间对齐及其相关性表示提高了VQA任务的训练效率。

英文摘要:

This work deals with the challenge of learning and reasoning over language and vision data for the related downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR). We design a novel cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under the supervision of a target task, which is more generalizable to unobserved data compared to merely reshaping the original representation space. In addition to modeling the relevance between the textual entities and visual entities, we model the higher-order relevance between entity relations in the text and object relations in the image. Our proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. The learned alignments of input spaces and their relevance representations by NLVR task boost the training efficiency of VQA task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑