arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19020cs.LGcs.AIcs.CL
收藏

通过主动重建学习检测语言模型训练数据

Learning to Detect Language Model Training Data via Active Reconstruction

University of Washington(华盛顿大学);Cornell University(康奈尔大学);Allen Institute for Artificial Intelligence(人工智能研究院)

Junjie Oscar Yin, John X. Morris, Vitaly Shmatikov, Sewon Min, Hannaneh Hajishirzi

AI总结:

本文提出主动数据重建攻击方法,通过强化学习主动诱导模型重建文本以检测语言模型训练数据,实验表明其在检测预训练、后训练和蒸馏数据方面表现优于现有方法。

摘要:

检测语言模型训练数据通常被框架为成员推断攻击(MIA)问题。然而,传统的MIA是在固定模型权重上被动操作,使用对数似然或文本生成。在本工作中,我们引入了主动数据重建攻击(ADRA),一种通过训练使模型重建给定文本的MIA家族。我们假设训练数据比非成员更可重建,且两者在可重建性上的差异可以用于成员推断。受发现强化学习(RL)可以增强已编码在权重中的行为的启发,我们利用在线强化学习,通过微调从目标模型初始化的策略,主动引发数据重建。为了有效利用RL进行MIA,我们设计了重建度量和对比奖励。所得到的算法,ADRA及其自适应变体ADRA+,在候选数据池中提高了重建和检测能力。实验表明,我们的方法在检测预训练、后训练和蒸馏数据方面优于现有MIA,平均改进率为10.7%。特别是,ADRA+在BookMIA预训练检测中比Min-K%++提高了18.8%,在AIME后训练检测中提高了7.6%。

英文摘要:

Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likelihoods or text generations. In this work, we introduce \textbf{Active Data Reconstruction Attack} (ADRA), a family of MIA that actively induces a model to reconstruct a given text through training. We hypothesize that training data are \textit{more reconstructible} than non-members, and the difference in their reconstructibility can be exploited for membership inference. Motivated by findings that reinforcement learning (RL) sharpens behaviors already encoded in weights, we leverage on-policy RL to actively elicit data reconstruction by finetuning a policy initialized from the target model. To effectively use RL for MIA, we design reconstruction metrics and contrastive rewards. The resulting algorithms, \textsc{ADRA} and its adaptive variant \textsc{ADRA+}, improve both reconstruction and detection given a pool of candidate data. Experiments show that our methods consistently outperform existing MIAs in detecting pre-training, post-training, and distillation data, with an average improvement of 10.7\% over the previous runner-up. In particular, \MethodPlus~improves over Min-K\%++ by 18.8\% on BookMIA for pre-training detection and by 7.6\% on AIME for post-training detection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑