Large language models (LLMs) have achieved impressive results in natural language processing but are prone to memorizing portions of their training data, which can compromise evaluation metrics, raise privacy concerns, and limit generalization. Traditional methods for detecting memorization rely on output probabilities or loss functions, often lacking precision due to confounding factors like common language patterns. In this paper, we introduce an analytical method that precisely detects memorization by examining neuron activations within the LLM. By identifying specific activation patterns that differentiate between memorized and not memorized tokens, we train classification probes that achieve near-perfect accuracy. The approach can also be applied to other mechanisms, such as repetition, as demonstrated in this study, highlighting its versatility. Intervening on these activations allows us to suppress memorization without degrading overall performance, enhancing evaluation integrity by ensuring metrics reflect genuine generalization. Additionally, our method supports large-scale labeling of tokens and sequences, crucial for next-generation AI models, improving training efficiency and results. Our findings contribute to model interpretability and offer practical tools for analyzing and controlling internal mechanisms in LLMs.
Unintended Memorization of Sensitive Information in Fine-Tuned Language Models
TUM University Hospital（慕尼黑工业大学医院）；Technical University of Munich (TUM)（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Computing, Imperial College London（伦敦帝国学院计算系）
Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；SynthLabs；Carnegie Mellon University（卡内基梅隆大学）
Unlocking Memorization in Large Language Models with Dynamic Soft Prompting
George Mason University（乔治梅森大学）；GE Healthcare（GE医疗）；University of Pittsburgh（匹兹堡大学）；William and Mary（威廉玛丽学院）；Southern University of Science and Technology（南方科技大学）；Florida State University（佛罗里达州立大学）
Reason to Rote: Rethinking Memorization in Reasoning
Department of ICS, Utrecht University（信息科学与计算系，乌特勒支大学）；MaiNLP, Center for Information and Language Processing, LMU Munich（MaiNLP，信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Saarland Informatics Campus, Saarland University（萨尔兰州信息科学校区，萨尔兰州大学）
Emergent and Predictable Memorization in Large Language Models
Booz Allen Hamilton（博思艾伦汉密尔顿）；EleutherAI；Yale University（耶鲁大学）；Ohio State University（俄亥俄州立大学）；Stability AI；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
Beyond Frequency: The Role of Redundancy in Large Language Model Memorization
Semyung University（_semyung大学）；University of Copenhagen（哥本哈根大学）；University of Washington（华盛顿大学）；SAIBD, Hefei University（合肥大学）；School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）
From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity
School of Intelligent Manufacturing Ecosystem, Xi’an Jiaotong-Liverpool University, People’s Republic of China（智能制造生态系统学院，西安交通大学-利物浦大学，中华人民共和国）；Department of Computer Science, University of Liverpool, United Kingdom（计算机科学系，利物浦大学，英国）；Faculty of Engineering and Technology, Sunway University, Malaysia（工程与技术学院，Sunway大学，马来西亚）；School of Robotics, Xi’an Jiaotong-Liverpool University, People’s Republic of China（机器人学院，西安交通大学-利物浦大学，中华人民共和国）
The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction
New York University（纽约大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；McGill University（麦吉尔大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Toronto（多伦多大学）；Vector Institute（矢量研究所）
Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications
Yale University（耶鲁大学）；Imperial College London（帝国理工学院）；University of Texas Health Science at Houston（德克萨斯大学休斯顿健康科学中心）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）；Vanderbilt University Medical Center（范德堡大学医学中心）；Vanderbilt University（范德堡大学）；National University of Singapore（新加坡国立大学）
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Laboratory of Transvascular Implantation Devices and TIDRI（血管植入设备国家重点实验室和TIDRI）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江医学影像人工智能重点实验室）；School of Data Science of Engineering, East China Normal University（华东师范大学工程数据科学学院）；Second Affiliated Hospital and Liangzhu Laboratory, Zhejiang University School of Medicine（浙江大学医学院第二附属医院和良渚实验室）；Alibaba Group（阿里巴巴集团）
KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
Huawei Poisson Laboratory（华为泊松实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Hong Kong University of Science and Technology（香港科技大学）
