（视觉）语言模型（(V)LMs）可泛化至表面共现之外：来自跨模态数一致的证据
(V)LMs generalize beyond surface co-occurrence: Evidence from cross-modal number agreement
浏览论文内容
中文总结 AI 辅助
该研究以(V)LMs为对象，通过跨模态泛化实验发现其可超越表面共现实现泛化，表现出与抽象兼容的行为，为语言模型具备抽象能力提供了证据。
中文摘要 AI 辅助
语言模型主要从共现中学习语法数，由此表现出频率效应——这有时被认为表明它们未学习抽象“规则”，而是依赖特定词汇项。仅用文本刺激测试泛化无法解决这一争论，因为分布线索（is/are、this/these）极易暴露数的信息。我们转而采用跨模态泛化作为工具，研究可接受视觉输入的视觉语言模型（VLMs）中的抽象能力，将诊断数的证据限制在非语言模态内。我们通过添加新嵌入并仅在学习过程中更新它们，向VLMs教授新名词对，对比仅由视觉线索诊断数的条件与由文本消除歧义的条件。在行为、表征动态和因果机制层面，我们在两种暴露条件下均发现了显著的跨模态泛化证据，且模型内部机制对语言线索与非语言线索条件的处理方式相似。这表明，诸如VLMs这类统计学习者可泛化至表面共现之外，表现出与真正抽象兼容的行为。
英文摘要
Language models learn about grammatical number primarily from co-occurrence, and show frequency effects as a result---sometimes taken to indicate that they do not learn abstract ``rules'', and are instead dependent on specific lexical items. Testing generalization with text stimuli alone cannot settle this debate, since distributional cues (is/are, this/these) easily give number away. We instead use cross-modal generalization as a tool to investigate abstractions in LMs that can also accept visual inputs (VLMs), restricting the evidence that diagnoses number to an extra-linguistic modality. We teach VLMs pairs of new nouns by adding new embeddings and only updating them during learning, comparing conditions where number is diagnosed by visual cues alone against ones where it is disambiguated by text. Across behavior, representational dynamics, and causal mechanisms, we find non-trivial evidence for cross-modal generalization across both exposure conditions, and that linguistic vs. extra-linguistic cue conditions are treated in similar ways in the internal mechanisms of the model. This suggests that statistical learners like VLMs can generalize beyond surface-level co-occurrence and show genuine abstraction-compatible behavior.
发表机构
- University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
- The University of Texas at Austin（德克萨斯大学奥斯汀分校）
机构由 AI 辅助整理，请以论文原文为准。
