大型语言模型中的事件知识：不可能与不太可能之间的差距
Event knowledge in large language models: the gap between the impossible and the unlikely
AI总结：
本文通过最小句对测试五个预训练LLMs的事件知识，发现它们能很好区分可能与不可能事件，但对可能与不太可能事件的区分一致性较差，揭示了分布式语言模式在事件知识习得中的优势与局限。
摘要：
语言语料库中的词语共现模式包含着数量惊人的概念知识。大型语言模型（LLMs）经过训练以根据上下文预测词语，利用这些模式在需要世界知识的多种语义任务上取得了令人印象深刻的表现。关于LLMs语义能力的一个重要但研究不足的问题是，它们是否习得了关于常见事件的概括性知识。在此，我们测试了五个预训练LLMs（从2018年的BERT到2023年的MPT）是否对施事-受事互动的合理描述赋予比对同一事件的最小差异不合理版本更高的似然。使用三组精心构建的最小句对（总计n=1,215），我们发现预训练LLMs拥有大量的事件知识，表现优于其他分布式语言模型。特别是，它们几乎总是对可能事件赋予比对不可能事件更高的似然（The teacher bought the laptop vs. The laptop bought the teacher）。然而，LLMs对可能事件与不太可能事件（The nanny tutored the boy vs. The boy tutored the nanny）表现出较不一致的偏好。在后续分析中，我们表明：(i) LLM评分同时受到合理性和表层句子特征的影响，(ii) LLM评分在句法变体（主动与被动结构）之间泛化良好，但在语义变体（同义句）之间泛化较差，(iii) 一些LLM错误反映了人类判断的模糊性，(iv) 句子合理性在LLM内部表征中充当组织维度。总体而言，我们的结果表明，事件知识的重要方面自然地从分布式语言模式中涌现，但也凸显了可能/不可能事件表征与可能/不太可能事件表征之间的差距。
英文摘要：
Word co-occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs' semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pre-trained LLMs (from 2018's BERT to 2023's MPT) assign higher likelihood to plausible descriptions of agent-patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n=1,215), we found that pre-trained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign higher likelihood to possible vs. impossible events (The teacher bought the laptop vs. The laptop bought the teacher). However, LLMs show less consistent preferences for likely vs. unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow-up analyses, we show that (i) LLM scores are driven by both plausibility and surface-level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
