Large Language Models (LLMs) have been reported to "leak" Personally Identifiable Information (PII), with successful PII reconstruction often interpreted as evidence of memorization. We propose a principled revision of memorization evaluation for LLMs, arguing that PII leakage should be evaluated under low lexical cue conditions, where target PII cannot be reconstructed through prompt-induced generalization or pattern completion. We formalize Cue-Resistant Memorization (CRM) as a cue-controlled evaluation framework and a necessary condition for valid memorization evaluation, explicitly conditioning on prompt-target overlap cues. Using CRM, we conduct a large-scale multilingual re-evaluation of PII leakage across 32 languages and multiple memorization paradigms. Revisiting reconstruction-based settings, including verbatim prefix-suffix completion and associative reconstruction, we find that their apparent effectiveness is driven primarily by direct surface-form cues rather than by true memorization. When such cues are controlled for, reconstruction success diminishes substantially. We further examine cue-free generation and membership inference, both of which exhibit extremely low true positive rates. Overall, our results suggest that previously reported PII leakage is better explained by cue-driven behavior than by genuine memorization, highlighting the importance of cue-controlled evaluation for reliably quantifying privacy-relevant memorization in LLMs.
PII-Bench: Evaluating Query-Aware Privacy Protection Systems
Institute of Fintech, Fudan University（复旦大学金融科技学院）；Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University（复旦大学数据科学上海重点实验室）；Laboratory of Data Analytics and Security, Fudan University（复旦大学数据安全分析实验室）
Unintended Memorization of Sensitive Information in Fine-Tuned Language Models
TUM University Hospital（慕尼黑工业大学医院）；Technical University of Munich (TUM)（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Computing, Imperial College London（伦敦帝国学院计算系）
RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；City University of Hong Kong（香港城市大学）；Virginia Tech（弗吉尼亚理工大学）；University of South Florida（南佛罗里达大学）；Rochester Institute of Technology（罗切斯特理工学院）
ProPILE: Probing Privacy Leakage in Large Language Models
NAVER AI Lab（NAVER人工智能实验室）；University of Luxembourg（卢森堡大学）；Parameter Lab（帕拉米特实验室）；Seoul National University（首尔国立大学）；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）
Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Kuaishou Technology（快手科技）；Meta AI；Monash University（墨尔本大学）；Fudan University（复旦大学）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）
Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearning
School of Software Technology, Zhejiang University（浙江大学软件技术学院）；Zhejiang University（浙江大学）；Hefei University of Technology（合肥工业大学）；Institute of Fundamental and Transdisciplinary Research, Zhejiang University（浙江大学基础与交叉学科研究院）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江区）区块链与数据安全研究院）；The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Zhejiang Key Laboratory of Big Data Intelligent Computing（浙江省大数据智能计算重点实验室）
AWED-PIPER: Agents, Web Applications & Expert Detectors for Personally Identifiable Information Protection & Fine-grained Named Entity Recognition across 36 languages for 6.6 Billion Speakers
Indian Institute of Technology Guwahati（印度理工学院古瓦哈蒂分校）
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs
The PII shown in the figure is synthetic and not real（合成的PII）；International Digital Economy Academy（国际数字经济学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；DataArc Tech Ltd（DataArc科技有限公司）
LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images
Center for Artificial Intelligence and Robotics (CAIRO)（人工智能与机器人中心（开罗））；Technical University of Applied Sciences Würzburg-Schweinfurt (THWS)（维尔茨堡-施韦因富特应用技术大学）；DataX
