Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance. A central challenge in assessing this risk is distinguishing prefix-specific memorization of training data from the generation of statistically common sequences. Existing approaches to measuring memorization often conflate these phenomena, labeling outputs as memorized even when they arise from generalization over common patterns. Counterfactual memorization and other related metrics \citep{zhang2023counterfactual, wang2025generalization, lesci2024causal} provide principled solutions, however, their reliance on retraining multiple baseline models or parsing through the training data makes them computationally impractical at scale.
This work introduces \emph{Prior-Aware memorization}, a theoretically grounded, lightweight and training-free criterion for identifying prefix-specific memorization in LLMs. The key idea is to evaluate whether a candidate suffix is strongly associated with its specific training prefix or whether it appears with high probability across many IID sampled sequences from the training data distribution due to statistical commonality. We correlate our metric with counterfactual memorization, and also evaluate it on the training corpora of two pre-trained models, LLaMA and OPT. Our results show that between 55\% and 90\% of sequences previously labeled as memorized fail our criterion and are consistent with statistical commonality
Estimating the Effects of Sample Training Orders for Large Language Models without Retraining
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学耿丽人工智能学院);Center for Data Science, Peking University(北京大学数据科学中心);School of Engineering Mathematics and Technology, University of Bristol(布里斯托大学工程数学与技术学院);Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学机器学习系)
Distinguishing between Normal and Cancer Cells Using Autoencoder Node Saliency
Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室)
基因表达谱已被广泛用于表征细胞对疾病的反应模式。随着数据变得可用,可扩展的学习工具包对于使用深度学习模型来处理大型数据集以模拟复杂的生物过程变得至关重要。我们提出了一个自动编码器来捕获从基因表达谱中恢复的非线性关系。自动编码器是一种使用人工神经网络的非线性降维技术,它学习未标记数据的隐藏表示。我们训练的自动编码器上收集的肿瘤样本从国家癌症研究所基因组数据共享,并获得一个广义和无监督的潜在表示。我们利用专注于HPC的深度学习工具包Livermore Big Artificial神经网络(LBANN)来高效地并行化训练算法,将计算时间从几个小时减少到几分钟。使用经过训练的自动编码器,我们生成了一个小数据集的潜在表示,其中包含各种肿瘤类型的正常细胞和癌细胞对。一种新的措施,称为自动编码器节点显着性(ANS)的介绍,以确定隐藏的节点,最好区分各种对细胞。我们比较我们的最佳分类节点的主成分分析和可视化的t分布随机邻居嵌入的结果。我们证明了自动编码器可以有效地为数据集中的多个学习任务提取不同的基因特征。