Recent advancements in vision models have greatly improved their ability to handle complex chart understanding tasks, like chart captioning and question answering. However, it remains challenging to assess how these models process charts. Existing benchmarks only roughly evaluate model performance without evaluating the underlying mechanisms, such as how models extract image embeddings. This limits our understanding of the model's ability to perceive fundamental graphical components. To address this, we introduce a novel evaluation framework to assess the graphical perception of image embedding models. For chart comprehension, we examine two main aspects of channel effectiveness: accuracy and discriminability of various visual channels. Channel accuracy is assessed through the linearity of embeddings, measuring how well the perceived magnitude aligns with the size of the stimulus. Discriminability is evaluated based on the distances between embeddings, indicating their distinctness. Our experiments with the CLIP model show that it perceives channel accuracy differently from humans and shows unique discriminability in channels like length, tilt, and curvature. We aim to develop this work into a broader benchmark for reliable visual encoders, enhancing models for precise chart comprehension and human-like perception in future applications.
Evaluating Graphical Perception Capabilities of Vision Transformers
Viscom Group, Institute of Media Informatics, Ulm University(Viscom集团,媒体信息学研究所,乌尔姆大学);ViRVIG Group, Universitat Politècnica de Catalunya - BarcelonaTech(ViRVIG集团,巴塞罗那技术大学-加泰罗尼亚理工大学);Ulm University(乌尔姆大学)
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Tongji University(同济大学);Nanjing University of Posts and Telecommunications(南京邮电大学);University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Nanjing University(南京大学)
RS3DBench: A Comprehensive Benchmark for 3D Spatial Perception in Remote Sensing
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Software College of Zhejiang University(浙江大学软件学院);Hangzhou City University(杭州市大学)
Cross-View Urban Sensing: Mapping Subjective Streetscape Perception via AlphaEarth Embeddings and Urban Context
School of Geospatial Engineering and Science, Sun Yat-sen University(中山大学地理空间工程学院);Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai)(南方海洋科学与工程广东省实验室(珠海));Key Laboratory of Comprehensive Observation of Polar Environment (Sun Yat-sen University), Ministry of Education(教育部极地环境综合观测重点实验室(中山大学))