基于 Jaccard 距离条件对比学习与上下文视觉增强的多模态实体链接增强
Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation
AI总结：
本文针对多模态实体链接中负样本选择不当的问题，提出 JD-CCL 方法利用元信息选取相似属性负样本以增强鲁棒性，并引入 CVaCPT 方法结合多视角合成图像与上下文文本增强视觉表示，在基准数据集上验证了有效性。
摘要：
以往关于多模态实体链接（MEL）的研究主要将对比学习作为主要目标。然而，这些研究在不加仔细考虑的情况下将批次中的其余部分作为负样本，存在利用简单特征的风险，并可能忽略使实体具有独特性的关键细节。在本研究中，我们提出了 JD-CCL（基于 Jaccard 距离的条件对比学习），这是一种旨在增强多模态实体链接模型匹配能力的新方法。JD-CCL 利用元信息来选择具有相似属性的负样本，使链接任务更具挑战性和鲁棒性。此外，为了解决提及与实体之间视觉模态内部变化所导致的局限性，我们引入了一种新方法 CVaCPT（上下文视觉辅助可控补丁变换）。它通过结合多视角合成图像和上下文文本表示来缩放和偏移补丁表示，从而增强视觉表示。在基准 MEL 数据集上的实验结果证明了我们方法的强大有效性。
英文摘要：
Previous research on multimodal entity linking (MEL) has primarily employed contrastive learning as the primary objective. However, using the rest of the batch as negative samples without careful consideration, these studies risk leveraging easy features and potentially overlook essential details that make entities unique. In this work, we propose JD-CCL (Jaccard Distance-based Conditional Contrastive Learning), a novel approach designed to enhance the ability to match multimodal entity linking models. JD-CCL leverages meta-information to select negative samples with similar attributes, making the linking task more challenging and robust. Additionally, to address the limitations caused by the variations within the visual modality among mentions and entities, we introduce a novel method, CVaCPT (Contextual Visual-aid Controllable Patch Transform). It enhances visual representations by incorporating multi-view synthetic images and contextual textual representations to scale and shift patch representations. Experimental results on benchmark MEL datasets demonstrate the strong effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
