通过双向生成对齐学习隐式实体-对象关系用于多模态命名实体识别
Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER
AI总结：
本文提出BGA-MNER方法，通过双向生成对齐和阶段细化上下文采样器学习隐式实体-对象关系，在推理时无需图像输入即可在多模态命名实体识别中取得最先进性能。
摘要：
多模态命名实体识别（MNER）所面临的挑战主要有两个方面：（1）弥合文本与图像之间的语义鸿沟，以及（2）将实体与图像中其关联的对象进行匹配。由于缺乏相应的标注，现有方法无法捕捉隐式的实体-对象关系。本文提出了一种名为BGA-MNER的双向生成对齐方法来解决这些问题。我们的BGA-MNER包含针对两种模态中实体显著内容的image2text和text2image生成。它联合优化双向重建目标，从而在这种直接且强力的约束下对齐隐式实体-对象关系。此外，图像-文本对通常包含不匹配的成分，这些成分对生成而言是噪声。我们提出了一种阶段细化上下文采样器，用于提取匹配的跨模态内容以进行生成。在两个基准数据集上的大量实验表明，我们的方法在推理阶段无需图像输入即可取得最先进的性能。
英文摘要：
The challenge posed by multimodal named entity recognition (MNER) is mainly two-fold: (1) bridging the semantic gap between text and image and (2) matching the entity with its associated object in image. Existing methods fail to capture the implicit entity-object relations, due to the lack of corresponding annotation. In this paper, we propose a bidirectional generative alignment method named BGA-MNER to tackle these issues. Our BGA-MNER consists of \texttt{image2text} and \texttt{text2image} generation with respect to entity-salient content in two modalities. It jointly optimizes the bidirectional reconstruction objectives, leading to aligning the implicit entity-object relations under such direct and powerful constraints. Furthermore, image-text pairs usually contain unmatched components which are noisy for generation. A stage-refined context sampler is proposed to extract the matched cross-modal content for generation. Extensive experiments on two benchmarks demonstrate that our method achieves state-of-the-art performance without image input during inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
