I2SRM:多模态信息抽取中的样本内与样本间关系建模
I2SRM: Intra- and Inter-Sample Relationship Modeling for Multimodal Information Extraction
AI总结:
提出I2SRM方法,通过样本内关系建模和AttnMixup样本间关系建模,在多模态信息抽取任务上取得领先性能。
摘要:
多模态信息抽取近年来受到研究关注,它需要聚合来自不同模态的表示。本文针对该任务提出了样本内与样本间关系建模(I2SRM)方法,该方法包含两个模块。首先,样本内关系建模模块在单个样本上操作,旨在学习有效的表示。来自文本和视觉模态的嵌入被移位,以弥合由不同的预训练语言模型和图像模型造成的模态差距。其次,样本间关系建模模块考虑多个样本之间的关系,并专注于捕捉交互。我们提出了一种AttnMixup策略,该策略不仅实现了样本间的协作,还通过数据增强提高了泛化能力。我们在多模态命名实体识别数据集Twitter-2015和Twitter-2017以及多模态关系抽取数据集MNRE上进行了大量实验。我们提出的I2SRM方法取得了有竞争力的结果,在Twitter-2015上F1分数为77.12%,在Twitter-2017上为88.40%,在MNRE上为84.12%。
英文摘要:
Multimodal information extraction is attracting research attention nowadays, which requires aggregating representations from different modalities. In this paper, we present the Intra- and Inter-Sample Relationship Modeling (I2SRM) method for this task, which contains two modules. Firstly, the intra-sample relationship modeling module operates on a single sample and aims to learn effective representations. Embeddings from textual and visual modalities are shifted to bridge the modality gap caused by distinct pre-trained language and image models. Secondly, the inter-sample relationship modeling module considers relationships among multiple samples and focuses on capturing the interactions. An AttnMixup strategy is proposed, which not only enables collaboration among samples but also augments data to improve generalization. We conduct extensive experiments on the multimodal named entity recognition datasets Twitter-2015 and Twitter-2017, and the multimodal relation extraction dataset MNRE. Our proposed method I2SRM achieves competitive results, 77.12% F1-score on Twitter-2015, 88.40% F1-score on Twitter-2017, and 84.12% F1-score on MNRE.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。