发表机构
Dublin City University; VNUHCM – University of Science（都柏林城市大学; 越南国立大学胡志明市科学大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究探讨语言模型下一词预测准确率与人类阅读理解认知信号的关系，基于两种信息度量为人类和模型生成回归器预测脑电信号的事件相关电位，发现只有意外性与语言处理电位相关，挑战了模型参数和预算增加必改善与人类语言处理收敛的假设。
AI 中文摘要
语言模型在给定前文的情况下预测序列中的下一个单词，人类在阅读理解中也有这种可预测性。神经科学研究表明，下一个单词的可预测性会影响大脑反应。虽然先进的语言模型在单词预测任务中的准确率与人类相近，但这引发了一个问题：更高的预测准确率是否意味着这些模型充分捕捉了与人类阅读理解相关的认知信号？我们基于两种信息度量为人类和语言模型生成回归器，以预测脑电记录中反映阅读中不同认知处理阶段的事件相关电位。我们认为对事件相关电位模式进行建模可以对各种语言模型在阅读过程中的认知合理性进行细粒度分析。结果表明只有意外性可能与语言处理的事件相关电位相关，特别是对于具有高语义内容的开放类单词。此外，我们的发现挑战了增加语言模型参数和计算预算会持续导致与类人语言处理更好收敛的假设。
英文摘要
Language models (LMs) are trained to excel at predicting the next word in the sequence given prior context, and humans also share this predictability in reading comprehension. Neuroscience research reveals that next-word predictability influences brain response, as recorded at millisecond resolution using electroencephalography (EEG). While our evidence indicates that advanced LMs achieve accuracies closely aligned with human performance at the next-word prediction task, this raises the question: Does higher prediction accuracy necessarily mean that these models adequately capture the cognitive signals associated with human reading comprehension? Here, we generate regressors for both humans and LMs based on two information measures, including top-1 prediction and surprisal, to predict event-related potential (ERP) elicited from EEG recordings which reflect different stages of cognitive processing during reading. We argue that modelling ERP patterns offers fine-grained analysis of the cognitive plausibility of various LMs during reading. Our results indicate that only surprisal potentially correlates with language-processing ERPs, especially for open-class words with high semantic content. Moreover, our findings challenge the assumption that scaling LMs with increased parameters and computational budgets will consistently lead to improved convergence with human-like linguistic processing.
