知道何时信任图像：可靠性感知的多模态实体对齐

中文摘要 AI 辅助

视觉模态（即图像）在多模态实体对齐（MMEA）中扮演关键角色。现有方法通常直接将图像与其他模态融合以对齐不同实体。尽管简单，但这些策略忽视了图像中潜在的噪声及其与对应实体在语义上的错位，导致融合效果次优和性能下降。针对这一问题，我们提出了一种新颖的可靠性感知框架用于MMEA（RA-MMEA），该框架评估视觉可靠性并自适应地改进不可靠的视觉表示，以实现稳健的实体对齐。其核心在于两个模块，包括依赖感知的视觉可靠性预测（DA-VRP）和稳定性正则化的视觉嵌入生成（SR-VEG）。前者旨在通过利用实体内的多模态依赖来估计图像的可靠性，而后者则专注于基于文本模态中编码的语义生成替代视觉表示，用于多模态融合。与当前方法相比，RA-MMEA能够为模态融合提供更可靠的视觉表示，从而提升性能。在大量实验中，RA-MMEA取得了最先进的结果，验证了可靠视觉模态对实体对齐的重要性以及RA-MMEA的有效性。代码和结果将公开发布。

英文摘要

The visual modality, i.e., images, plays a key role in multi-modal entity alignment (MMEA). Existing approaches often directly fuse the image with other modalities to align different entities. Although simple, such strategies overlook the potential noise in the images and their semantic misalignment with corresponding entities, resulting in suboptimal fusion and degraded performance. Addressing this, we propose a novel Reliability-Aware framework for MMEA (RA-MMEA), which assesses visual reliability and adaptively improves unreliable visual representations for robust entity alignment. The core lies in two modules, including dependency-aware visual reliability prediction (DA-VRP) and stability-regularized visual embedding generation (SR-VEG). The former aims to estimate the reliability of an image by leveraging multi-modal dependency within the entity, while the latter focuses on producing alternative visual representation conditioned on semantics encoded in textual modalities for multi-modal fusion. Compared to current methods, RA-MMEA enables more reliable visual representations for modality fusion, thereby improving performance. In extensive experiments, RA-MMEA achieves state-of-the-art results, verifying the importance of reliable visual modality for entity alignment and the effectiveness of RA-MMEA. The code and results will be released.

发表机构

• University of North Texas（北德克萨斯大学）
• Purdue University（普渡大学）
• Meta Inc（Meta公司）

机构由 AI 辅助整理，请以论文原文为准。