arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11792cs.AIcs.CL
收藏

通过推理的结构收敛检测RLVR训练数据

Detecting RLVR Training Data via Structural Convergence of Reasoning

Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院)

Hongbo Zhang, Yue Yang, Jianhao Yan, Guangsheng Bao, Yue Zhang, Yue Zhang

AI总结:

通过推理结构收敛检测RLVR训练数据,提出Min-kNN距离方法有效区分训练数据与未见数据。

摘要:

可验证奖励的强化学习(RLVR)是训练现代推理模型的核心,但未披露的训练数据引发了基准污染的担忧。与预训练方法不同,RLVR通过自我生成的推理轨迹的奖励反馈来微调模型,这使得传统基于似然的检测方法效果不佳。我们表明,RLVR会引发一种独特的行为特征:在RLVR训练期间遇到的提示会产生更僵化和相似的生成,而未见过的提示则保留更大的多样性。我们引入了Min-kNN距离,这是一种简单的黑盒检测器,通过为给定提示采样多个完成并计算k个最小最近邻编辑距离的平均值来量化这种坍缩。Min-kNN距离不需要访问参考模型或令牌概率。在多个RLVR训练的推理模型上的实验表明,Min-kNN距离可靠地区分了RL见过的例子和未见过的例子,并且在现有成员推断和RL污染检测基线中表现更优。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-$k$NN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the $k$ smallest nearest-neighbor edit distances. Min-$k$NN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-$k$NN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑