识别大语言模型中的预训练数据：一种基于神经元激活的检测框架
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework
AI总结：
本文提出NA-PDD算法，通过分析LLM中训练与非训练数据的差异化神经元激活模式来检测预训练数据，并构建时间无偏基准CCNewsPDD，在多个基准和模型上显著超越现有方法。
摘要：
大语言模型（LLMs）的性能与其训练数据密切相关，而这些数据可能包含受版权保护的材料或私人信息，从而引发法律和伦理问题。此外，LLMs还因数据集污染和内化偏见而受到批评。为解决这些问题，人们提出了预训练数据检测（PDD）任务，以识别特定数据是否包含在LLM的预训练语料库中。然而，现有的PDD方法通常依赖预测置信度和损失等表层特征，导致性能平庸。为改进这一点，我们提出了NA-PDD，这是一种新颖的算法，分析LLMs中训练数据与非训练数据之间差异化的神经元激活模式。该方法基于以下观察：在LLM推理过程中，这些数据类型会激活不同的神经元。我们还引入了CCNewsPDD，这是一个在时间上无偏的基准，采用严格的数据变换以确保训练数据与非训练数据之间具有一致的时间分布。我们的实验表明，NA-PDD在三个基准和多个LLMs上显著优于现有方法。
英文摘要：
The performance of large language models (LLMs) is closely tied to their training data, which can include copyrighted material or private information, raising legal and ethical concerns. Additionally, LLMs face criticism for dataset contamination and internalizing biases. To address these issues, the Pre-Training Data Detection (PDD) task was proposed to identify if specific data was included in an LLM's pre-training corpus. However, existing PDD methods often rely on superficial features like prediction confidence and loss, resulting in mediocre performance. To improve this, we introduce NA-PDD, a novel algorithm analyzing differential neuron activation patterns between training and non-training data in LLMs. This is based on the observation that these data types activate different neurons during LLM inference. We also introduce CCNewsPDD, a temporally unbiased benchmark employing rigorous data transformations to ensure consistent time distributions between training and non-training data. Our experiments demonstrate that NA-PDD significantly outperforms existing methods across three benchmarks and multiple LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
