The training of modern large language models (LLMs) takes place in a regime where most training examples are seen only a few times by the model during the course of training. What does a model remember about such examples seen only a few times during training and how long does that memory persist in the face of continuous training with new examples? Here, we investigate these questions through simple recognition, recall, and retention experiments with LLMs. In recognition experiments, we ask if the model can distinguish the seen example from a novel example; in recall experiments, we ask if the model can correctly recall the seen example when cued by a part of it; and in retention experiments, we periodically probe the model's memory for the original examples as the model is trained continuously with new examples. We find that a single exposure is generally sufficient for a model to achieve near perfect accuracy even in very challenging recognition experiments. We estimate that the recognition performance of even small language models easily exceeds human recognition performance reported in similar experiments with humans (Shepard, 1967). Achieving near perfect recall takes more exposures, but most models can do it in just 3 exposures. The flip side of this remarkable capacity for fast learning is that precise memories are quickly overwritten: recall performance for the original examples drops steeply over the first 10 training updates with new examples, followed by a more gradual decline. Even after 100K updates, however, some of the original examples are still recalled near perfectly. A qualitatively similar retention pattern has been observed in human long-term memory retention studies before (Bahrick, 1984). Finally, recognition is much more robust to interference than recall and memory for natural language sentences is generally superior to memory for stimuli without structure.
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall
School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院,中国);Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用国家重点实验室(东南大学),中华人民共和国教育部,中国);College of Design and Innovation, Tongji University, China(同济大学设计与创新学院,中国)
Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks
Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Intel Labs(英特尔实验室);University of Texas at Austin(德克萨斯大学奥斯汀分校);Princeton University(普林斯顿大学);Princeton Neuroscience Institute(普林斯顿神经科学研究所);New York University(纽约大学)
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Peng Cheng Laboratory(鹏城实验室);Huazhong University of Science and Technology(华中科技大学);Xiamen University(厦门大学);The Hong Kong University of Science and Technology, Guangzhou(香港科技大学(广州));School of Biomedical Engineering, Tsinghua University(清华大学生物医学工程学院)
MoM: Linear Sequence Modeling with Mixture-of-Memories
Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Fudan University(复旦大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Chinese University of Hong Kong(香港中文大学)