arXiv 2606.079962026-06-06cs.CLcs.AI

MC-PDD: 面向黑盒大语言模型的掩码语料级预训练数据检测

MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models

University of Macau（澳门大学）；Macau Millennium College（澳门万人大学）；BoardWare Information System Limited（博纬信息系统有限公司）

Kaixin Lan, Mu You, Tao Fang, Binkai Ou, Lidia S. Chao, Derek F. Wong

备注：: The manuscript consists of 10 pages formatted in the IEEE/ACM two-column style

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出MC-PDD方法，通过掩码特定token并利用LLM预测缺失内容，比较候选语料与参考非成员语料的预测命中率差异，以黑盒方式检测预训练数据，性能与现有方法相当。

摘要：

预训练是大语言模型（LLM）发展的基础，然而预训练数据的不透明性使模型分析复杂化，并引发伦理、法律和公平性问题。因此，检测特定数据集是否在预训练中使用至关重要。现有最先进方法通常依赖于访问模型概率分布，因此不适用于仅提供输入输出接口的闭源LLM。为解决这一限制，我们引入了掩码语料级预训练数据检测（MC-PDD），这是一种受掩码语言建模范式启发的新方法。MC-PDD在每段文本中掩码高度特定的token，并提示LLM预测缺失内容。然后，它评估候选语料与参考非成员语料之间的预测命中率差异是否具有统计显著性。基于此比较，MC-PDD确定候选文本是否可能包含在模型的预训练数据中。实验结果表明，在三个数据集上，对于开源和闭源LLM，预训练数据和未见数据之间的预测命中率存在明显且一致的差异。尽管在更严格的黑盒设置下运行，MC-PDD仍实现了与现有检测方法相当的性能。我们的方法仅需使用标准API访问即可实现模型审计和数据版权验证等实际应用。接受后，我们将公开发布代码和数据集。

英文摘要：

Pretraining is fundamental to the development of Large Language Models (LLMs), yet the opacity of pretraining data complicates model analysis and raises ethical, legal, and fairness concerns. Detecting whether specific datasets were used during pretraining is, therefore, critical. Existing state-of-the-art methods typically rely on access to model probability distributions, making them unsuitable for closed-source LLMs that provide only input-output interfaces. To address this limitation, we introduce Masked Corpus-level Pretraining Data Detection (MC-PDD), a novel method inspired by the masked language modeling paradigm. MC-PDD masks highly specific tokens in each text and prompts the LLM to predict the missing content. It then assesses whether the difference in prediction hit rates between a candidate corpus and a reference non-member corpus is statistically significant. Based on this comparison, MC-PDD determines whether the candidate texts were likely included in the model's pretraining data. Experimental results demonstrate clear and consistent differences in prediction hit rates between pretrained and unseen data across three datasets, for both open-source and closed-source LLMs. Despite operating under a stricter black-box setting, MC-PDD achieves performance comparable to existing detection methods. Our approach enables practical applications such as model auditing and data copyright verification using only standard API access. Upon acceptance, we will publicly release the code and datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。