探索多模态Web文档上统一的以视觉为中心的对比替代方案
Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents
AI总结：
针对现有对比视觉语言模型处理复杂Web文档能力有限的问题，提出以视觉为中心的统一对比学习框架VC2L，通过像素空间渲染和片段级对比学习实现多模态建模，在多个基准上取得优于或媲美CLIP类模型的性能。
摘要：
CLIP等对比视觉语言模型通过从对齐的图像-文本对中学习，已在广泛的多模态任务中展现出优异性能。然而，它们处理复杂的真实世界Web文档的能力仍然有限，尤其是在文本与图像交错、松散对齐或以视觉形式嵌入的场景中。为应对这些挑战，我们提出了以视觉为中心的对比学习（Vision-Centric Contrastive Learning，VC2L），这是一个使用单个视觉Transformer对文本、图像及其组合进行建模的统一框架。VC2L完全在像素空间中运行，通过将所有输入（无论是文本、视觉还是组合输入）渲染为图像，从而无需OCR、文本分词或模态融合策略。为捕捉多模态Web文档中复杂的跨模态关系，VC2L采用片段级对比学习目标，对齐连续的多模态片段，利用文档的内在连贯性，无需显式配对的图像-文本数据。为评估该方法的有效性，我们引入了三个检索基准：AnyCIR、SeqCIR和CSR，分别用于评估跨模态检索、细粒度序列理解以及对未见过数据的泛化能力。实验结果表明，VC2L在我们提出的基准以及M-BEIR和MTEB等已建立的数据集上，与CLIP风格的模型相比，取得了具有竞争力或更优的性能。这些发现凸显了多模态Web数据作为对比学习宝贵训练资源的潜力，并阐明了统一的以视觉为中心的方法在多模态表示学习中的可扩展性。代码和模型可在以下网址获取：https://github.com/showlab/VC2L。
英文摘要：
Contrastive vision-language models such as CLIP have demonstrated strong performance across a wide range of multimodal tasks by learning from aligned image-text pairs. However, their ability to handle complex, real-world web documents remains limited, particularly in scenarios where text and images are interleaved, loosely aligned, or embedded in visual form. To address these challenges, we propose Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer. VC2L operates entirely in pixel space by rendering all inputs, whether textual, visual, or combined, as images, thus eliminating the need for OCR, text tokenization, or modality fusion strategy. To capture complex cross-modal relationships in multimodal web documents, VC2L employs a snippet-level contrastive learning objective that aligns consecutive multimodal segments, leveraging the inherent coherence of documents without requiring explicitly paired image-text data. To assess the effectiveness of this approach, we introduce three retrieval benchmarks, AnyCIR, SeqCIR, and CSR, designed to evaluate cross-modal retrieval, fine-grained sequential understanding, and generalization to unseen data, respectively. Empirical results show that VC2L achieves competitive or superior performance compared to CLIP-style models on both the proposed benchmarks and established datasets such as M-BEIR and MTEB. These findings underscore the potential of multimodal web data as a valuable training resource for contrastive learning and illustrate the scalability of a unified, vision-centric approach for multimodal representation learning. Code and models are available at: https://github.com/showlab/VC2L.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
