视觉编码器能学会看箭头吗？
Can Visual Encoder Learn to See Arrows?
AI总结：
本研究通过对比学习训练视觉编码器，消除文本和位置偏差，使其在图表边识别上超越CLIP及零样本GPT-4o和LLaVA-Mistral，为提升VLM图表理解提供了有效路径。
摘要：
图表是通过边（线条或箭头）来展示关系的视觉表示，广泛应用于工业和科学交流中。尽管识别图表对于视觉语言模型（VLMs）理解领域特定知识至关重要，但近期研究表明，许多VLM无法识别图像中的边。我们假设这些失败源于对文本和位置偏差的过度依赖，这阻碍了VLM学习明确的边特征。基于这一想法，我们通过实验探究了VLM中的图像编码器是否能够通过在一个边既不受文本也不受位置偏差影响的图表数据集上进行训练来学习边的表示。为此，我们在一个人工生成的图表-标题数据集上进行对比学习以训练图像编码器，并在三项任务上评估其与图表相关的特征：探测、图像检索和图像描述生成。我们的结果显示，微调后的模型在所有任务上均优于预训练的CLIP，并在图像描述生成任务上超越了零样本的GPT-4o和LLaVA-Mistral。这些发现证实，消除文本和位置偏差有助于VLM实现准确的边识别，为推进图表理解提供了一条有前景的路径。
英文摘要：
The diagram is a visual representation of a relationship illustrated with edges (lines or arrows), which is widely used in industrial and scientific communication. Although recognizing diagrams is essential for vision language models (VLMs) to comprehend domain-specific knowledge, recent studies reveal that many VLMs fail to identify edges in images. We hypothesize that these failures stem from an over-reliance on textual and positional biases, preventing VLMs from learning explicit edge features. Based on this idea, we empirically investigate whether the image encoder in VLMs can learn edge representation through training on a diagram dataset in which edges are biased neither by textual nor positional information. To this end, we conduct contrastive learning on an artificially generated diagram--caption dataset to train an image encoder and evaluate its diagram-related features on three tasks: probing, image retrieval, and captioning. Our results show that the finetuned model outperforms pretrained CLIP in all tasks and surpasses zero-shot GPT-4o and LLaVA-Mistral in the captioning task. These findings confirm that eliminating textual and positional biases fosters accurate edge recognition in VLMs, offering a promising path for advancing diagram understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
