语言模型并未学习语言的物理表现形式
Language Models Don't Learn the Physical Manifestation of Language
AI总结:
本文通过 H-Test 系列任务实证表明,仅语言模型缺乏语言的视觉-听觉物理表现理解能力,即使使用思维链、少样本示例或更强模型也接近随机水平,揭示了纯语言知识在缺乏感官经验时的根本局限。
摘要:
我们认为,仅使用语言的模型并未学习语言的物理表现形式。我们通过一系列称为 H-Test 的任务,对语言的视觉-听觉特性进行了实证研究。这些任务凸显了人类语言理解与 LLM 在缺乏感官输入条件下的语言理解之间的根本差距。为支持我们的假设,实验表明:1. 刻意推理(Chain-of-Thought)、2. few-shot 示例、或 3. 来自同一模型家族的更强 LLM(LLaMA 2 13B -> LLaMA 2 70B)对 H-Test 性能均无显著影响。我们引入哲学案例 Mary——她在感官剥夺的环境中了解世界——作为理解仅语言模型如何了解世界的有用概念框架(Jackson, 1986)。我们的实验表明,一些最强的专有 LLM 的准确率仍接近 50% 的随机基线,凸显了在缺乏感官经验的情况下所获得的语言知识的局限性。我们的代码和数据可在 <github.com/brucewlee/h-test> 获取。
英文摘要:
We argue that language-only models don't learn the physical manifestation of language. We present an empirical investigation of visual-auditory properties of language through a series of tasks, termed H-Test. These tasks highlight a fundamental gap between human linguistic understanding and the sensory-deprived linguistic understanding of LLMs. In support of our hypothesis, 1. deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3. stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) has no significant effect on H-Test performance. We bring in the philosophical case of Mary, who learns about the world in a sensory-deprived environment as a useful conceptual framework to understand how language-only models learn about the world (Jackson, 1986). Our experiments show that some of the strongest proprietary LLMs stay near random chance baseline accuracy of 50%, highlighting the limitations of linguistic knowledge acquired in the absence of sensory experience. Our code and data are available at <github.com/brucewlee/h-test>.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。