结构感知对比学习用于多模态模型的图表理解
Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Models
AI总结：
本文提出结构感知对比学习方法，通过专门的损失函数提升多模态模型对图表图像的理解能力，在图像-文本匹配和视觉问答任务中取得显著改进。
摘要：
多模态模型，如对比语言-图像预训练（CLIP）模型，在对齐视觉和语言表示方面取得了显著成功。然而，这些模型在应用于专门的视觉领域时存在局限性，例如图表，它们编码了结构化的符号信息，与自然图像不同。在本文中，我们提出了一种新的训练范式，专门设计以增强视觉语言模型对图表图像的理解。我们的方法使用
英文摘要：
Multimodal models, such as the Contrastive Language-Image Pre-training (CLIP) model, have demonstrated remarkable success in aligning visual and linguistic representations. However, these models exhibit limitations when applied to specialised visual domains, such as diagrams, which encode structured, symbolic information distinct from that of natural imagery. In this paper, we introduce a novel training paradigm explicitly designed to enhance the comprehension of diagrammatic images within vision-language models. Our approach uses ``hard'' samples for our proposed contrastive learning that incorporates two specialised loss functions that leverage the inherent structural properties of diagrams. By integrating these objectives into model training, our method enables models to develop a more structured and semantically coherent understanding of diagrammatic content. We empirically validate our approach on a benchmark dataset of flowcharts, as a representative class of diagrammatic imagery, demonstrating substantial improvements over standard CLIP and conventional hard negative CLIP learning paradigms for both image-text matching and visual question answering tasks. Our findings underscore the significance of tailored training strategies for specialised tasks and contribute to advancing diagrammatic understanding within the broader landscape of vision-language integration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
