DocumentCLIP：在重排文档中链接图表与正文
DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents
AI总结：
本文提出显著性感知对比学习框架DocumentCLIP，通过收集大型Wikipedia数据集进行预训练，解决文档内部多图文对齐问题，在有监督和零样本设置下均优于基线模型。
摘要：
视觉-语言预训练模型通过理解图像与文本之间的对齐关系，在支持多媒体应用方面取得了巨大成功。虽然现有的视觉-语言预训练模型主要侧重于理解与单段文本相关联的单张图像，但它们往往忽略了文档内部的对齐关系，这种对齐关系由多句话和多张图像组成。在这项工作中，我们提出了DocumentCLIP，这是一个显著性感知的对比学习框架，旨在使视觉-语言预训练模型能够理解文档内图像与较长文本之间的交互作用。我们的模型对于现实世界的多模态文档理解是有益的，例如新闻文章、杂志、产品描述，这些内容在语言和视觉上都更加丰富。据我们所知，我们是第一个通过对比学习探索多模态文档内部链接的研究。此外，我们收集了一个大型Wikipedia数据集用于预训练，该数据集提供了各种主题和结构。实验表明，DocumentCLIP不仅在有监督设置中优于最先进的基线模型，而且在人工评估后，在自然环境中也实现了最佳的零样本性能。我们的代码可在 https://github.com/FuxiaoLiu/DocumentCLIP 获取。
英文摘要：
Vision-language pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding single image associated with a single piece of text, they often ignore the alignment at the intra-document level, consisting of multiple sentences with multiple images. In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents. Our model is beneficial for the real-world multimodal document understanding like news article, magazines, product descriptions, which contain linguistically and visually richer content. To the best of our knowledge, we are the first to explore multimodal intra-document links by contrastive learning. In addition, we collect a large Wikipedia dataset for pretraining, which provides various topics and structures. Experiments show DocumentCLIP not only outperforms the state-of-the-art baselines in the supervised setting, but also achieves the best zero-shot performance in the wild after human evaluation. Our code is available at https://github.com/FuxiaoLiu/DocumentCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
