情景记忆是否有助于缩小句法对比敏感性中的词汇频率差距？使用检索增强型语言模型的测试
Does Episodic Memory Help Close the Lexical Frequency Gap in Sensitivity to Syntactic Contrasts? A Test Using Retrieval-Augmented Language Models
- ENS（法国高等师范学院）
- Université PSL（巴黎文理大学）
- EHESS（高等社会科学研究院）
- CNRS（法国国家科学研究中心）
- Boston University（波士顿大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究利用检索增强型语言模型检验情景记忆可缩小句法对比测试中的词汇频率差距，发现检索增强能缩小高低频项的性能差距，且结构信息对有效检索至关重要。
AI中文摘要：
语法知识及其经验测试通常被认为对表达式中词汇项的频率具有鲁棒性，但基于神经网络的语法性模型对词汇频率表现出高度敏感性。我们利用互补学习系统理论，检验了“对词汇频率的鲁棒性可通过海马体情景记忆机制产生”这一假设，该机制能快速编码和检索特定经验，使学习者在处理稀有模式时可利用这些经验。我们将检索增强型语言模型作为此类情景记忆机制的实例（具体为k近邻语言模型，其通过显式实例存储增强参数模型），并测试这种增强是否有助于缩小普通语言模型在句法对比测试中表现出的词汇频率差距。使用频率分层测试项的句法对比，我们发现检索增强缩小了高频与低频项之间的性能差距，与情景记忆补偿弱参数表示的假设一致。该益处适用于不同句法现象，以及在儿童现实数据和大规模数据上预训练的模型。此外，我们表明结构信息对有效检索至关重要，而仅语义相似性几乎无益处。尽管这些是支持我们假设的有前景的概念验证结果，但频率差距仅被缩小而非完全消除。基于我们的分析，我们提出对检索实例的优先重加权、针对结构信息的更好表示与检索策略，以及存储和检索的灵活配置，作为改进语言模型中情景记忆实现的有前景未来方向。
英文摘要：
Grammatical knowledge and how it is empirically tested are typically considered robust to the frequency of the lexical items in the expressions. However, neural network-based models of grammaticality exhibit high sensitivity to lexical frequency. We draw upon Complementary Learning Systems theory to test the hypothesis that robustness to lexical frequency can arise via a hippocampal episodic memory mechanism, which enables rapid encoding and retrieval of specific experiences and allows learners to leverage them when processing rare patterns. We use retrieval-augmented language models as an instantiation of such an episodic memory mechanism (specifically, $k$-nearest-neighbor language models that augment parametric models with explicit instance storage), and test whether this augmentation helps close the lexical frequency gap that vanilla language models exhibit in syntactic contrast tests. Using syntactic contrasts with frequency-stratified test items, we find that retrieval augmentation narrows the performance gap between high- and low-frequency items, consistent with episodic memory compensating for weak parametric representations. This benefit is consistent across different syntactic phenomena and across models pretrained on child-realistic and large-scale data. Additionally, we show that structural information is critical for effective retrieval, whereas semantic similarity alone provides little benefit. While these are promising proof-of-concept results supporting our hypothesis, the frequency gap is narrowed rather than fully closed. Based on our analyses, we propose preferential reweighting of retrieved instances, better representations and retrieval strategies for structural information, and flexible configurations of storage and retrieval as promising future directions for improving the implementation of episodic memory in language models.
