arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.14883cs.CL
收藏

语言 in vivo 与 in silico:规模固然重要,但由于不可穿透的语义指代,更大的语言模型仍无法像人类那样理解语言

Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference

University of Pavia(帕维亚大学);Humboldt-Universitat zu Berlin(柏林洪堡大学);Universitat Autònoma de Barcelona(巴塞罗那自治大学);Institució Catalana de Recerca i Estudis Avançats (ICREA)(加泰罗尼亚研究与高级研究所)

Vittoria Dentella, Fritz Guenther, Evelina Leivada

AI总结:

本研究探讨了模型规模扩大是否能弥补LLMs与人类在语法判断任务上的差异,发现尽管更大的模型(如ChatGPT-4)总体准确率略高于人类,但在回答稳定性和语法敏感度上仍与人类存在本质差异,仅靠规模扩展难以解决由不可穿透语义指代引起的根本问题。

摘要:

理解语言的限制是大型语言模型(LLMs)作为自然语言理论的先决条件。LLMs 在某些语言任务中的表现与人类存在定量和定性的差异,然而,这些差异是否可以通过模型规模来弥补仍有待确定。本研究调查了模型缩放的关键作用,确定规模的增加是否能弥补人类与模型之间的这些差异。我们在一个语法判断任务上测试了来自不同家族的三个 LLMs(Bard,1370亿参数;ChatGPT-3.5,1750亿参数;ChatGPT-4,1.5万亿参数),该任务包含照应、中心嵌套、比较级和否定极性。收集了 N=1,200 次判断,并对其准确性、稳定性以及在重复呈现提示时准确性的提高进行了评分。表现最好的 LLM(ChatGPT-4)的结果与 n=80 名人类在相同刺激下的结果进行了比较。我们发现,人类的总体准确性低于 ChatGPT-4(分别为 76% 和 80% 的准确率),但这是因为 ChatGPT-4 仅在一个任务条件(即语法句子)上优于人类。此外,ChatGPT-4 在回答中比人类更摇摆不定(出现摇摆答案的可能性分别为 12.5% 和 9.6%)。因此,尽管增加模型规模可能会带来更好的性能,但 LLMs 对(不合)语法的敏感度仍与人类不同。仅靠缩放似乎可能但不太可能解决这个问题。我们通过比较 in vivo 和 in silico 的语言学习来解释这些结果,识别出三个关键差异,涉及 证据的类型、 刺激的匮乏,以及 由于不可穿透的语言指代而出现的语义幻觉。

英文摘要:

Understanding the limits of language is a prerequisite for Large Language Models (LLMs) to act as theories of natural language. LLM performance in some language tasks presents both quantitative and qualitative differences from that of humans, however it remains to be determined whether such differences are amenable to model size. This work investigates the critical role of model scaling, determining whether increases in size make up for such differences between humans and models. We test three LLMs from different families (Bard, 137 billion parameters; ChatGPT-3.5, 175 billion; ChatGPT-4, 1.5 trillion) on a grammaticality judgment task featuring anaphora, center embedding, comparatives, and negative polarity. N=1,200 judgments are collected and scored for accuracy, stability, and improvements in accuracy upon repeated presentation of a prompt. Results of the best performing LLM, ChatGPT-4, are compared to results of n=80 humans on the same stimuli. We find that humans are overall less accurate than ChatGPT-4 (76% vs. 80% accuracy, respectively), but that this is due to ChatGPT-4 outperforming humans only in one task condition, namely on grammatical sentences. Additionally, ChatGPT-4 wavers more than humans in its answers (12.5% vs. 9.6% likelihood of an oscillating answer, respectively). Thus, while increased model size may lead to better performance, LLMs are still not sensitive to (un)grammaticality the same way as humans are. It seems possible but unlikely that scaling alone can fix this issue. We interpret these results by comparing language learning in vivo and in silico, identifying three critical differences concerning (i) the type of evidence, (ii) the poverty of the stimulus, and (iii) the occurrence of semantic hallucinations due to impenetrable linguistic reference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑