时间上下文恢复驱动长上下文语言模型中的情景式顺序记忆
Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models
浏览论文内容
中文总结 AI 辅助
研究长上下文语言模型中情景式顺序记忆的机制，通过时间顺序记忆任务及新数据集，发现模型呈现与人类相同的距离效应，且性能依赖一维时间编码，由单个注意力头在检索时恢复，支持时间上下文恢复是重要机制。
中文摘要 AI 辅助
人类情景记忆支持对长时间尺度上展开的经历的检索，但其潜在计算机制因人类长期记忆实验中机制可及性有限而仍有争议。长上下文语言模型可能为揭示驱动此类检索的合理计算机制提供途径。本文通过时间顺序记忆任务研究语言模型是否以及如何捕捉情景记忆的核心行为特征。利用基于长篇小说记忆的人类行为新数据集，发现模型在此任务上呈现出与人类相同的特征距离效应。接着通过长上下文机制可解释性分析揭示模型解决该任务的方式，发现模型性能依赖于一维时间编码，由单个时间恢复注意力头在检索时恢复。这些发现支持时间上下文恢复是语言模型中情景式时间顺序记忆的重要机制，为长期情景记忆的时间方面在人工和生物系统中如何实现提供新见解。
英文摘要
Human episodic memory supports the retrieval of experiences that unfold over extended timescales, yet the computational mechanisms underlying this ability remain debated due to the limited mechanistic accessibility in long-term memory experiments in humans. Long-context LLMs may offer promising ways to reveal plausible computational mechanisms that drive this type of retrieval. Here, we investigate whether and how LLMs capture the core behavioral signatures of episodic memory via a temporal order memory task. Using a new dataset of human behavior based on memory of a full-length novel, we show that models exhibit the same characteristic distance effect observed in humans on this task. We next apply long-context mechanistic interpretability analyses to uncover how models solve this task, and find that model performance relies on a one-dimensional temporal code that is reinstated during retrieval by a single time-reinstatement attention head. These findings support temporal context reinstatement as an important mechanism for episodic-like temporal-order memory in LLMs, offering new insights into how temporal aspects of long-term episodic memory may be instantiated in both artificial and biological systems.
