视觉象征性挑战：在手语形式-意义映射上评估视觉-语言模型
The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping
AI总结：
本文提出一个新颖的视频基准测试，用于评估视觉-语言模型在手语形式-意义映射上的表现，通过心理语言学测量来评估三种任务：语音学手语形式预测、透明度和渐进象征性评分，并发现模型在语音形式预测上表现较好但整体仍低于人类表现。
摘要：
象征性，即语言形式与意义之间的相似性，在手语中普遍存在，为视觉 grounding 提供了自然的测试环境。对于视觉-语言模型（VLMs），挑战在于从动态的人类运动中恢复这种本质的映射，而非静态上下文。我们引入了视觉象征性挑战，一个新颖的基于视频的基准测试，将心理语言学测量适应于评估 VLMs 在三个任务上的表现：（i）语音学手语形式预测（例如，手形、位置），（ii）透明度（从视觉形式推断意义），以及（iii）渐进象征性评分。我们评估了13种最先进的VLMs在零样本和少样本设置下在荷兰手语上的表现，并将其与人类基线进行比较。在语音形式预测上，VLMs恢复了一些手形和位置细节，但表现仍低于人类；在透明度上，它们与人类基线相差甚远；只有顶级模型与人类象征性评分有中等相关性。有趣的是，语音形式预测能力更强的模型更能与人类象征性判断相关联，表明它们对视觉基础结构有共同的敏感性。我们的发现验证了这些诊断任务，并推动了以人类为中心的信号和具身学习方法，用于建模象征性和改进多模态模型中的视觉 grounding。
英文摘要：
Iconicity, the resemblance between linguistic form and meaning, is pervasive in signed languages, offering a natural testbed for visual grounding. For vision-language models (VLMs), the challenge is to recover such essential mappings from dynamic human motion rather than static context. We introduce the Visual Iconicity Challenge, a novel video-based benchmark that adapts psycholinguistic measures to evaluate VLMs on three tasks: (i) phonological sign-form prediction (e.g., handshape, location), (ii) transparency (inferring meaning from visual form), and (iii) graded iconicity ratings. We assess 13 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands and compare them to human baselines. On phonological form prediction, VLMs recover some handshape and location detail but remain below human performance; on transparency, they are far from human baselines; and only top models correlate moderately with human iconicity ratings. Interestingly, models with stronger phonological form prediction correlate better with human iconicity judgment, indicating shared sensitivity to visually grounded structure. Our findings validate these diagnostic tasks and motivate human-centric signals and embodied learning methods for modelling iconicity and improving visual grounding in multimodal models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
