基于多模态对比学习的表征链接
Linking Representations with Multimodal Contrastive Learning
AI总结：
针对跨数据集实体链接任务中传统字符串匹配方法精度不足、单模态方法易受OCR噪声或模态能力局限的问题，提出多模态对比学习框架CLIPPINGS，在历史企业记录链接任务上性能显著优于传统方法，且多模态预训练能提升单模态推理的效果。
摘要：
许多应用需要在不同数据集间匹配个人、企业或地点实体。目前广泛使用的方法（尤其是社会科学领域的方法）尚未引入深度学习，记录链接通常采用字符串匹配技术实现。此外，现有方法未利用文档固有的多模态属性。在历史记录链接场景中，文档通常由光学字符识别（OCR）转录，存在噪声干扰：仅依赖OCR文本进行链接可能因噪声失效，仅依赖图像裁剪块进行链接也可能因视觉模型缺乏语言理解能力（例如无法识别企业名称的缩写或其他不同写法）而失败。\n为发挥多模态学习的优势，本研究提出CLIPPINGS（Contrastively LInking Pooled Pre-trained Embeddings，对比链接池化预训练嵌入）框架。该框架首先通过在文档图像及其对应OCR文本上执行对比语言-图像预训练，实现对称视觉双编码器与语言双编码器的对齐；随后通过对比学习构建度量空间，使同一实例的池化图文嵌入与同类（如同一企业或地点）嵌入距离相近，与不同类嵌入距离较远。实体链接任务被转化为基于多模态嵌入的最近邻检索问题，以此完成数据匹配。\n在20世纪中期日本企业跨金融文档链接任务上，CLIPPINGS的性能大幅优于常用字符串匹配方法。仅通过对齐企业名称图像裁剪块与其对应OCR文本的嵌入训练得到的纯自监督模型，性能同样优于主流字符串匹配方法。值得注意的是，经多模态预训练的单视觉编码器性能优于单模态预训练的单视觉编码器，这表明即使推理时仅有一种模态可用于链接，多模态预训练仍能发挥显著作用。
英文摘要：
Many applications require linking individuals, firms, or locations across datasets. Most widely used methods, especially in social science, do not employ deep learning, with record linkage commonly approached using string matching techniques. Moreover, existing methods do not exploit the inherently multimodal nature of documents. In historical record linkage applications, documents are typically noisily transcribed by optical character recognition (OCR). Linkage with just OCR'ed texts may fail due to noise, whereas linkage with just image crops may also fail because vision models lack language understanding (e.g., of abbreviations or other different ways of writing firm names). To leverage multimodal learning, this study develops CLIPPINGS (Contrastively LInking Pooled Pre-trained Embeddings). CLIPPINGS aligns symmetric vision and language bi-encoders, through contrastive language-image pre-training on document images and their corresponding OCR'ed texts. It then contrastively learns a metric space where the pooled image-text embedding for a given instance is close to embeddings in the same class (e.g., the same firm or location) and distant from embeddings of a different class. Data are linked by treating linkage as a nearest neighbor retrieval problem with the multimodal embeddings. CLIPPINGS outperforms widely used string matching methods by a wide margin in linking mid-20th century Japanese firms across financial documents. A purely self-supervised model - trained only by aligning the embeddings for the image crop of a firm name and its corresponding OCR'ed text - also outperforms popular string matching methods. Fascinatingly, a multimodally pre-trained vision-only encoder outperforms a unimodally pre-trained vision-only encoder, illustrating the power of multimodal pre-training even if only one modality is available for linking at inference time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
