arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.20845cs.CVcs.HCcs.LG
收藏

使用通道有效性评估图像嵌入模型的图形感知

Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness

Seoul National University(首尔大学)

Soohyun Lee, Minsuk Chang, Seokhyeon Park, Jinwook Seo

AI总结:

本文提出一个评估图像嵌入模型图形感知的新颖框架,通过通道准确性和可辨别性两方面评估视觉通道有效性,发现CLIP模型在长度、倾斜和曲率等通道上具有与人类不同的感知能力。

摘要:

视觉模型的最新进展极大地提升了它们处理复杂图表理解任务的能力,如图表描述和问答。然而,评估这些模型如何处理图表仍然具有挑战性。现有的基准仅粗略评估模型性能,而未评估其底层机制,例如模型如何提取图像嵌入。这限制了我们对模型感知基本图形组件能力的理解。为了解决这个问题,我们引入了一个新颖的评估框架来评估图像嵌入模型的图形感知能力。对于图表理解,我们检查了通道有效性的两个主要方面:各种视觉通道的准确性和可辨别性。通道准确性通过嵌入的线性度来评估,测量感知量级与刺激大小的一致程度。可辨别性基于嵌入之间的距离进行评估,表明它们的独特性。我们对 CLIP 模型的实验表明,它对通道准确性的感知与人类不同,并在长度、倾斜和曲率等通道上显示出独特的可辨别性。我们旨在将这项工作发展为一个更广泛的可靠视觉编码器基准,增强模型在未来应用中精确图表理解和类人感知的能力。

英文摘要:

Recent advancements in vision models have greatly improved their ability to handle complex chart understanding tasks, like chart captioning and question answering. However, it remains challenging to assess how these models process charts. Existing benchmarks only roughly evaluate model performance without evaluating the underlying mechanisms, such as how models extract image embeddings. This limits our understanding of the model's ability to perceive fundamental graphical components. To address this, we introduce a novel evaluation framework to assess the graphical perception of image embedding models. For chart comprehension, we examine two main aspects of channel effectiveness: accuracy and discriminability of various visual channels. Channel accuracy is assessed through the linearity of embeddings, measuring how well the perceived magnitude aligns with the size of the stimulus. Discriminability is evaluated based on the distances between embeddings, indicating their distinctness. Our experiments with the CLIP model show that it perceives channel accuracy differently from humans and shows unique discriminability in channels like length, tilt, and curvature. We aim to develop this work into a broader benchmark for reliable visual encoders, enhancing models for precise chart comprehension and human-like perception in future applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑