arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.11047cs.CV
收藏

文本到图像检索的跨模态一致性

Cross-Modal Coherence for Text-to-Image Retrieval

Rutgers University(罗格斯大学);University of Pittsburgh(匹兹堡大学)

Malihe Alikhani, Fangda Han, Hareesh Ravi, Mubbasir Kapadia, Vladimir Pavlovic, Matthew Stone

摘要:

常见的图像-文本联合理解技术假定图像和相关文本可以普遍地由单个隐式模型表征。然而,共现图像和文本可以以不同的方式联系在一起,明确地对其建模可以提高当前联合理解模型的性能。在本文中,我们训练一个跨模态的连贯性模型的文本到图像检索任务。我们的分析表明,与图像-文本连贯性关系训练的模型可以检索图像最初配对的目标文本更经常比连贯性不可知模型。我们还表明,通过人类的评价,图像检索所提出的一致性感知模型是首选的一致性不可知的基线由一个巨大的利润。我们的研究结果提供了不同的方式,沟通和连贯关系的作用,在捕捉常识推理的文本和图像的见解。

英文摘要:

Common image-text joint understanding techniques presume that images and the associated text can universally be characterized by a single implicit model. However, co-occurring images and text can be related in qualitatively different ways, and explicitly modeling it could improve the performance of current joint understanding models. In this paper, we train a Cross-Modal Coherence Modelfor text-to-image retrieval task. Our analysis shows that models trained with image--text coherence relations can retrieve images originally paired with target text more often than coherence-agnostic models. We also show via human evaluation that images retrieved by the proposed coherence-aware model are preferred over a coherence-agnostic baseline by a huge margin. Our findings provide insights into the ways that different modalities communicate and the role of coherence relations in capturing commonsense inferences in text and imagery.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑