基于跨模态检索与合成的多模态关系抽取
Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis
AI总结：
本文针对多模态关系抽取中复杂关系识别不足的问题，提出基于对象、句子和整幅图像检索文本与视觉证据，并合成对象级、图像级和句子级信息进行跨模态推理，实验表明其显著优于现有最先进模型。
摘要：
多模态关系抽取（MRE）是基于句子-图像对的上下文识别两个实体之间语义关系的任务。现有的检索增强方法主要侧重于对检索到的文本知识进行建模，但这可能无法准确识别复杂关系。为了改进预测，本研究提出基于对象、句子和整幅图像来检索文本和视觉证据。我们进一步开发了一种新方法，对对象级、图像级和句子级信息进行合成，以便在同一模态和不同模态之间进行更好的推理。大量实验和分析表明，所提出的方法能够有效地选择和比较跨模态证据，并显著优于最先进的模型。
英文摘要：
Multimodal relation extraction (MRE) is the task of identifying the semantic relationships between two entities based on the context of the sentence image pair. Existing retrieval-augmented approaches mainly focused on modeling the retrieved textual knowledge, but this may not be able to accurately identify complex relations. To improve the prediction, this research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image. We further develop a novel approach to synthesize the object-level, image-level, and sentence-level information for better reasoning between the same and different modalities. Extensive experiments and analyses show that the proposed method is able to effectively select and compare evidence across modalities and significantly outperforms state-of-the-art models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
