arXiv 2510.092592025-10cs.CLcs.AIcs.LG

在大型语言模型后训练强化学习中检测数据污染

Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models

School of Computer Science, Peking University（北京大学计算机学院）；New H3C Technologies Co., Ltd（新华H3C技术有限公司）

Yongding Tao, Tian Wang, Yihong Dong, Huanyu Liu, Kechi Zhang, Xiaolong Hu, Ge Li

备注：: Accepted to ICLR 2026. Code is available at https://github.com/yongding-tao/RL-Data-Contamination

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Self-Critique方法，通过分析LLM后强化学习阶段输出熵分布的变化来检测数据污染，实验表明其在多个模型和污染任务中显著优于基线方法，AUC提升达30%。

摘要：

数据污染对大型语言模型（LLMs）的可靠评估构成重大威胁。当基准样本可能意外出现在训练集中时，会破坏报告性能的有效性。尽管已开发出预训练和监督微调阶段的检测方法，但在日益重要的后训练强化学习（RL）阶段仍存在关键研究空白。由于RL后训练对LLM推理至关重要，该范式中缺乏专门的污染检测方法成为关键漏洞。为此，我们进行了首次系统研究RL后训练中的数据检测，并提出Self-Critique。我们的方法受到关键观察的启发：在RL阶段后，LLM的输出熵分布倾向于坍缩为高度特定和稀疏的模式。Self-Critique探测底层策略坍缩，即模型收敛到狭窄的推理路径，导致熵减少。为促进此研究，我们还引入了RL-MIA基准，用于模拟这种特定污染场景。广泛实验表明，Self-Critique在多个模型和污染任务中显著优于基线方法，AUC提升高达30%。尽管现有方法在RL阶段污染检测上接近随机猜测，我们的方法使检测成为可能。

英文摘要：

Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. As RL post-training becomes pivotal for advancing LLM reasoning, the absence of specialized contamination detection methods in this paradigm presents a critical vulnerability. To address this, we conduct the first systematic study of data detection within RL post-training scenario and propose Self-Critique. Our method is motivated by a key observation: after RL phase, the output entropy distribution of LLMs tends to collapse into highly specific and sparse modes. Self-Critique probes for the underlying policy collapse, i.e., the model's convergence to a narrow reasoning path, which causes this entropy reduction. To facilitate this research, we also introduce RL-MIA, a benchmark constructed to simulate this specific contamination scenario. Extensive experiments show that Self-Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving an AUC improvement of up to 30%. Whereas existing methods are close to a random guess for RL-phase contamination, our method makes detection possible.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文