ColorBench：视觉语言模型能否看见并理解多彩世界？一个面向颜色感知、推理与鲁棒性的综合基准
ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
AI总结：
本文提出ColorBench基准，系统评估32个视觉语言模型在颜色感知、推理与鲁棒性上的表现，揭示规模定律仍成立、语言模型作用更大、思维链推理可提升颜色理解等发现，为多模态AI的颜色理解研究提供基础工具。
摘要：
颜色在人类感知中扮演重要角色，并且通常为视觉推理提供关键线索。然而，目前尚不清楚视觉语言模型（VLMs）是否以及如何像人类一样感知、理解和利用颜色。本文提出了ColorBench，这是一个精心设计的创新基准，旨在评估VLMs在颜色理解方面的能力，包括颜色感知、推理和鲁棒性。通过构建一系列多样化且基于真实应用的测试场景，ColorBench评估这些模型如何感知颜色、从基于颜色的线索中推断含义，以及在不同颜色变换下保持稳定表现的能力。通过对32个具有不同语言模型和视觉编码器的VLMs进行广泛评估，本文揭示了一些此前未被发现的发现：（i）规模定律（更大的模型更好）在ColorBench上仍然成立，同时语言模型比视觉编码器发挥更重要的作用。（ii）然而，不同模型之间的性能差距相对较小，表明现有VLMs在很大程度上忽视了颜色理解。（iii）思维链（CoT）推理提高了颜色理解的准确性和鲁棒性，尽管这些任务以视觉为中心。（iv）在ColorBench上，VLMs确实利用了颜色线索，但在某些任务中这些线索也可能误导模型。这些发现突显了当前VLMs的关键局限性，并强调了增强颜色理解能力的必要性。我们的ColorBench可以作为推进多模态AI实现人类水平颜色理解研究的基础工具。
英文摘要：
Color plays an important role in human perception and usually provides critical clues in visual reasoning. However, it is unclear whether and how vision-language models (VLMs) can perceive, understand, and leverage color as humans. This paper introduces ColorBench, an innovative benchmark meticulously crafted to assess the capabilities of VLMs in color understanding, including color perception, reasoning, and robustness. By curating a suite of diverse test scenarios, with grounding in real applications, ColorBench evaluates how these models perceive colors, infer meanings from color-based cues, and maintain consistent performance under varying color transformations. Through an extensive evaluation of 32 VLMs with varying language models and vision encoders, our paper reveals some undiscovered findings: (i) The scaling law (larger models are better) still holds on ColorBench, while the language model plays a more important role than the vision encoder. (ii) However, the performance gaps across models are relatively small, indicating that color understanding has been largely neglected by existing VLMs. (iii) CoT reasoning improves color understanding accuracies and robustness, though they are vision-centric tasks. (iv) Color clues are indeed leveraged by VLMs on ColorBench but they can also mislead models in some tasks. These findings highlight the critical limitations of current VLMs and underscore the need to enhance color comprehension. Our ColorBenchcan serve as a foundational tool for advancing the study of human-level color understanding of multimodal AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
