arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.06408cs.CL
收藏

人类和语言模型在预测重复文本时存在分歧

Humans and language models diverge when predicting repeating text

UT Austin(德克萨斯大学奥斯汀分校);Intel Labs(英特尔实验室)

Aditya R. Vaidya, Javier Turek, Alexander G. Huth

AI总结:

研究发现人类与GPT-2在预测重复文本时表现出现分歧,通过在中间层特定注意力头添加幂律近因偏置,使模型表现更接近人类。

摘要:

在下一词预测任务上训练的语言模型已被证明能准确建模人类在词汇预测和阅读速度上的行为。与这些发现相反,我们提出了一个人类与LMs表现出现分歧的场景。我们收集了一个数据集,包含人类对五种由重复文本片段组成的刺激材料的下一词预测。人类和GPT-2 LM的预测在文本片段首次呈现时高度一致,但当记忆(或上下文学习)开始发挥作用时,它们的性能迅速出现分歧。我们将这种分歧的原因追溯到中间层的特定注意力头。向这些注意力头添加幂律近因偏置后,得到的模型表现与人类更为相似。我们希望这一场景能激发未来的研究,使LMs更接近人类行为。

英文摘要:

Language models that are trained on the next-word prediction task have been shown to accurately model human behavior in word prediction and reading speed. In contrast with these findings, we present a scenario in which the performance of humans and LMs diverges. We collected a dataset of human next-word predictions for five stimuli that are formed by repeating spans of text. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory (or in-context learning) begins to play a role. We traced the cause of this divergence to specific attention heads in a middle layer. Adding a power-law recency bias to these attention heads yielded a model that performs much more similarly to humans. We hope that this scenario will spur future work in bringing LMs closer to human behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑