弱连接，强信号：通过独立令牌采样实现扩散大语言模型的高效训练数据检测
Weak Ties, Strong Signals: Efficient Training Data Detection in Diffusion LLMs via Independent Token Sampling
浏览论文内容
中文总结 AI 辅助
针对扩散大语言模型训练数据检测，提出独立令牌采样（ITS）框架，通过弱依赖令牌选择降低近似误差，显著提升检测性能。
中文摘要 AI 辅助
扩散大语言模型（dLLMs）为自回归模型提供了一种有吸引力的替代方案，但它们在去噪过程中可能暴露敏感的训练数据。检测这种使用情况具有挑战性，因为dLLMs缺乏因果架构的高效单遍概率分解。现有方法依赖于随机掩码，在有限的查询预算下获得可处理的逐令牌检测信号，但未能控制掩码令牌之间的依赖关系。我们证明，这种逐令牌近似引入了非负的结构性估计误差，该误差在理论上由掩码令牌之间的累积条件互信息（CMI）表征，并可能掩盖微妙的记忆信号。这一见解表明，可靠的检测需要内部依赖较弱的掩码令牌集。为了避免直接估计令牌组合上的CMI所带来的高昂成本，我们提出了独立令牌采样（ITS），这是一个查询高效的框架，使用基于注意力的成对依赖代理来近似CMI感知的选择标准。ITS进一步结合了促进多样性的策略，以提高跨采样轮的令牌覆盖率，从而产生受依赖引起的近似误差影响较小的聚合逐令牌信号。在多个数据集上的实验表明，ITS在不同模型和数据集上始终优于最先进的基线，在ArXiv数据集上实现了0.18的AUC提升，同时在有限的查询预算下保持了强劲的性能。代码可在该https URL获取。
英文摘要
Diffusion large language models (dLLMs) offer a compelling alternative to autoregressive models, yet they may expose sensitive training data during denoising. Detecting such usage is challenging because dLLMs lack the efficient one-pass probability decomposition of causal architectures. Existing methods rely on random masking to obtain tractable token-wise detection signals under limited query budgets, but fail to control dependencies among masked tokens. We demonstrate that this token-wise approximation introduces a non-negative structural estimation error, which is theoretically characterized by the cumulative conditional mutual information (CMI) among masked tokens and can obscure subtle memorization signals. This insight suggests that reliable detection requires masked token sets with weak internal dependency. To avoid the prohibitive cost of directly estimating CMI over token combinations, we propose \textit{Independent Token Sampling} (ITS), a query-efficient framework that uses an attention-derived pairwise dependency proxy to approximate the CMI-aware selection criterion. ITS further incorporates a diversity-promoting strategy to improve token coverage across sampling rounds, yielding aggregated token-wise signals that are less affected by dependency-induced approximation error. Experiments on multiple datasets show that ITS consistently outperforms state-of-the-art baselines across different models and datasets, achieving an AUC improvement of 0.18 on the ArXiv dataset while maintaining strong performance under limited query budgets. The code is available at https://github.com/Chrisqcwx/DLLM-MIA .
发表机构
- Tsinghua University（清华大学）
- Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
机构由 AI 辅助整理，请以论文原文为准。
