神经语言模型与人类增量处理的针对性评估
A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans
AI总结：
通过十六个句法测试套件对比人类与四种神经语言模型的增量处理，发现模型虽在方向上与人类一致，但系统低估了句法违例导致的处理难度差异幅度。
摘要：
我们通过在一系列结构现象中收集十六个不同句法测试套件的逐词反应时数据，对人类与神经语言模型的增量处理进行了有针对性的、扩大规模的比较。人类反应时数据来自一种名为 Interpolated Maze 任务的新型在线实验范式。我们将人类反应时与四种当代语言模型的逐词概率进行比较，这些模型具有不同的架构，并在不同规模的数据集上进行了训练。我们发现，在许多现象中，人类和语言模型在不合语法的句子区域都表现出处理难度增加，人类和模型的“准确率”得分（参照 Marvin and Linzen(2018)）大致相等。然而，尽管语言模型的输出在方向上与人类一致，我们表明模型系统性地低估了合语法与不合语法句子之间增量处理难度差异的幅度。具体而言，当模型遇到句法违例时，它们未能准确预测在人类数据中观察到的更长反应时。这些结果对当代语言模型在句法违例敏感性方面是否正在接近人类水平的性能提出了质疑。
英文摘要：
We present a targeted, scaled-up comparison of incremental processing in humans and neural language models by collecting by-word reaction time data for sixteen different syntactic test suites across a range of structural phenomena. Human reaction time data comes from a novel online experimental paradigm called the Interpolated Maze task. We compare human reaction times to by-word probabilities for four contemporary language models, with different architectures and trained on a range of data set sizes. We find that across many phenomena, both humans and language models show increased processing difficulty in ungrammatical sentence regions with human and model `accuracy' scores (a la Marvin and Linzen(2018)) about equal. However, although language model outputs match humans in direction, we show that models systematically under-predict the difference in magnitude of incremental processing difficulty between grammatical and ungrammatical sentences. Specifically, when models encounter syntactic violations they fail to accurately predict the longer reaction times observed in the human data. These results call into question whether contemporary language models are approaching human-like performance for sensitivity to syntactic violations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
