图像与文本学习的极限：视觉语言模型与具身场景理解
The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
AI总结：
研究探讨图像与文本学习在人类场景理解中的局限性，通过对比18个视觉语言模型与2000名人类观察者在15项任务中的表现，发现模型在具身认知任务中存在显著缺陷，表明分布学习不足以实现基于 affordance 的场景理解。
摘要：
什么信息足以学习人类对场景的全面理解？分布假说认为语言和图像的统计共现能捕捉视觉认知背后的概念知识。视觉语言模型（VLMs）在大规模配对文本-图像语料上训练，但缺乏具身经验，是检验分布假说的理想测试。我们报告了两项实验，比较18个VLMs生成的描述与2000多名人类观察者在15项高阶场景理解任务中的表现，涵盖常识、 affordances、感官体验、情感反应和未来预测。由于许多任务缺乏真实答案，我们开发了人类校准余弦距离（HCD）度量标准，衡量VLM输出与人类响应分布的相似性，按内人变异性缩放。在实验1中，VLMs在常识任务中接近人类水平，但在需要提示工程的affordance任务中表现出显著缺陷，且不随新模型发布而改善。在实验2中，我们测试了六个解释该affordance差距的机理假说，发现缺陷是结构性而非风格性，且未通过提供显式空间信息解决。语料分析揭示图像描述语料库中包含稀疏的agent-addressed affordance语言，与Gricean解释为何具身知识可能在语言中系统性被低估一致。共同发现表明，仅从图像和文本进行分布学习不足以实现基于affordance的场景理解，暗示某些人类视觉认知维度可能需要agent-centered、三维经验，而照片或描述无法编码。
英文摘要：
What information is sufficient to learn the full richness of human scene understanding? The distributional hypothesis holds that the statistical co-occurrence of language and images captures the conceptual knowledge underlying visual cognition. Vision-language models (VLMs) are trained on massive paired text-image corpora but lack embodied experience, making them an ideal test of the distributional hypothesis. We report two experiments comparing descriptions generated by 18 VLMs to those of over 2000 human observers across 15 high-level scene understanding tasks, spanning general knowledge, affordances, sensory experiences, affective responses, and future prediction. Because many tasks lack ground truth answers, we developed a Human-Calibrated Cosine Distance (HCD) metric that measures VLM output similarity to the distribution of human responses, scaled by within-human variability. In Experiment 1, VLMs approached human-level performance on general knowledge tasks, but showed a robust deficit for affordance tasks that resisted prompt engineering and did not improve with newer model releases. In Experiment 2, we tested six mechanistic hypotheses for explaining this affordance gap, finding that the deficit was structural rather than stylistic and was not resolved by providing explicit spatial information. Corpus analyses revealed that image captioning datasets contain sparse agent-addressed affordance language, consistent with Gricean accounts of why embodied knowledge may be systematically underrepresented in language. Together, these findings suggest that distributional learning from images and text is insufficient for affordance-based scene understanding, implying that some dimensions of human visual cognition may require the kind of agent-centered, three-dimensional experience that no photograph or caption can encode.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
