发表机构
Universitat de València; Image Processing Lab（瓦伦西亚大学; 图像处理实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出基于Hues and Cues游戏的评估框架，对比162个CVLMs与人类的颜色感知对齐，发现其在抽象领域偏离人类基线，筛选的预训练数据集可缓解错位。
AI 中文摘要
评估对比视觉语言模型（CVLMs）与人类的感知对齐程度时，传统基准通常忽略了细粒度的语义和文化细微差别，存在局限。本研究提出一种新颖的评估框架，利用桌面游戏Hues and Cues的游戏化离散色彩空间，将游戏的480个颜色单元映射到CIE xy色度图，在涵盖7个语义类别的精心筛选的100个词汇上计算经验感知距离。为了合理设定模型性能的上下文，我们通过定制数字界面收集325名人类观察者的密集颜色关联数据集，采用留一法（LOO）交叉验证建立了经验预期误差下界——人类一致性基线。我们评估了多个架构家族和预训练数据集的162个模型，以评估它们的语义颜色接地能力。结果表明，CVLMs虽能成功复现人类的认知偏差，例如对具体物理对象（如食物和植物）的理想化记忆颜色，但在抽象、主观和流行文化领域却系统性地偏离人类基线。我们在严重错位的概念中识别出两种不同的失败模式：语义分类错误和系统性不确定性崩溃为默认蓝色坐标。此外，我们发现精心筛选的预训练数据集在缓解这些严重错位方面，比庞大的未筛选语料库有效得多。最终，本研究强调，尽管当前CVLMs具备广泛的分类能力，但仍无法捕捉人类颜色记忆的细微本地化共识，凸显了游戏化任务在揭示模型潜在偏差方面的价值。相关数据和代码已公开，可用于测试其他指标。
英文摘要
Evaluating the perceptual alignment between Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances. In this work, we propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues. By mapping the board's 480 color cells to the CIE xy chromaticity diagram, we calculate empirical perceptual distances across a carefully curated 100-word vocabulary spanning seven semantic categories. To properly contextualize model performance, we establish an empirical lower bound of expected error-the Human Consistency baseline-calculated via Leave-One-Out (LOO) cross-validation on a dense dataset of color associations collected from 325 human observers through a custom digital interface. We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding. Our results demonstrate that while CVLMs successfully replicate human cognitive biases, such as idealized memory colors for concrete physical referents (e.g., food and plants), they systematically diverge from the human baseline in abstract, subjective, and pop-culture domains. We identify two distinct failure modes in severely misaligned concepts: semantic misclassification and a systematic uncertainty collapse into a default blue coordinate. Furthermore, we reveal that highly curated pre-training datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments. Ultimately, this work highlights that despite their broad categorization capabilities, current CVLMs still fail to capture the nuanced, localized consensus of human color memory, emphasizing the value of gamified tasks in exposing underlying model biases. The data and code are publicly available to test other metrics.
Comments19 pages, 14 figures
