基于查询式实体对象Transformer的多模态关系三元组抽取
Multimodal Relational Triple Extraction with Query-based Entity Object Transformer
AI总结：
针对现有多模态关系抽取需预给定实体对象的局限，本文提出多模态实体-对象关系三元组抽取新任务，构建对应数据集，并提出带选择性注意力的查询式模型QEOT，可同时完成三类任务，性能超基线8.06%达SOTA。
摘要：
多模态关系抽取对于构建灵活且符合真实场景的知识图谱至关重要。现有研究聚焦于抽取不同模态中已存在实体对的关系类型，例如一个实体在文本中、另一个在图像中，但这类方法需要预先给定实体和对象，成本高昂且不具备实用性。针对这一局限，本文提出一项新任务：多模态实体-对象关系三元组抽取，目标是从图文对中抽取所有形如（实体跨度、关系、对象区域）的三元组。为支撑该研究，我们对包含21种关系类型的多模态关系抽取数据集MORE进行修改，构建了一个包含20264个三元组的新数据集，平均每个图文对包含5.75个三元组。此外，本文提出QEOT——一种带有选择性注意力机制的查询式模型，用于动态探索文本与视觉信息的交互与融合。特别地，该方法可通过一组查询同时完成实体抽取、关系分类和对象检测，适用于下游应用，且能减少流水线式方法带来的误差累积。大量实验结果表明，本文方法比现有基线模型性能高出8.06%，达到当前最优水平。
英文摘要：
Multimodal Relation Extraction is crucial for constructing flexible and realistic knowledge graphs. Recent studies focus on extracting the relation type with entity pairs present in different modalities, such as one entity in the text and another in the image. However, existing approaches require entities and objects given beforehand, which is costly and impractical. To address the limitation, we propose a novel task, Multimodal Entity-Object Relational Triple Extraction, which aims to extract all triples (entity span, relation, object region) from image-text pairs. To facilitate this study, we modified a multimodal relation extraction dataset MORE, which includes 21 relation types, to create a new dataset containing 20,264 triples, averaging 5.75 triples per image-text pair. Moreover, we propose QEOT, a query-based model with a selective attention mechanism, to dynamically explore the interaction and fusion of textual and visual information. In particular, the proposed method can simultaneously accomplish entity extraction, relation classification, and object detection with a set of queries. Our method is suitable for downstream applications and reduces error accumulation due to the pipeline-style approaches. Extensive experimental results demonstrate that our proposed method outperforms the existing baselines by 8.06% and achieves state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
