arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11349cs.CLcs.AI
收藏

语言模型并未学习语言的物理表现形式

Language Models Don't Learn the Physical Manifestation of Language

University of Pennsylvania(宾夕法尼亚大学)

Bruce W. Lee, JaeHyuk Lim

AI总结:

本文通过 H-Test 系列任务实证表明,仅语言模型缺乏语言的视觉-听觉物理表现理解能力,即使使用思维链、少样本示例或更强模型也接近随机水平,揭示了纯语言知识在缺乏感官经验时的根本局限。

摘要:

我们认为,仅使用语言的模型并未学习语言的物理表现形式。我们通过一系列称为 H-Test 的任务,对语言的视觉-听觉特性进行了实证研究。这些任务凸显了人类语言理解与 LLM 在缺乏感官输入条件下的语言理解之间的根本差距。为支持我们的假设,实验表明:1. 刻意推理(Chain-of-Thought)、2. few-shot 示例、或 3. 来自同一模型家族的更强 LLM(LLaMA 2 13B -> LLaMA 2 70B)对 H-Test 性能均无显著影响。我们引入哲学案例 Mary——她在感官剥夺的环境中了解世界——作为理解仅语言模型如何了解世界的有用概念框架(Jackson, 1986)。我们的实验表明,一些最强的专有 LLM 的准确率仍接近 50% 的随机基线,凸显了在缺乏感官经验的情况下所获得的语言知识的局限性。我们的代码和数据可在 <github.com/brucewlee/h-test> 获取。

英文摘要:

We argue that language-only models don't learn the physical manifestation of language. We present an empirical investigation of visual-auditory properties of language through a series of tasks, termed H-Test. These tasks highlight a fundamental gap between human linguistic understanding and the sensory-deprived linguistic understanding of LLMs. In support of our hypothesis, 1. deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3. stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) has no significant effect on H-Test performance. We bring in the philosophical case of Mary, who learns about the world in a sensory-deprived environment as a useful conceptual framework to understand how language-only models learn about the world (Jackson, 1986). Our experiments show that some of the strongest proprietary LLMs stay near random chance baseline accuracy of 50%, highlighting the limitations of linguistic knowledge acquired in the absence of sensory experience. Our code and data are available at <github.com/brucewlee/h-test>.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑