DVD：一种检测大规模语言模型评估中变体污染的稳健方法
DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation
AI总结：
DVD通过分析生成分布的方差检测大规模语言模型评估中的变体污染，优于多种现有检测方法。
摘要：
评估大规模语言模型（LLMs）时，越来越受到变体污染的干扰：训练语料库包含语义等价但词法或语法上有所改变的测试项目变体。与逐字泄漏不同，这些改写或结构变换的变体逃避了基于采样一致性或困惑度的现有检测器，从而通过记忆而非真正的推理来提高基准分数。我们正式化了这一问题，并引入DVD（通过生成分布方差检测变体污染），一种单样本检测器，该检测器建模了由温度采样诱导的局部输出分布。我们的关键见解是受污染的项目会在记忆一致性状态和扰动漂移状态之间交替，导致低概率标记的合成难度异常高方差；未受污染的项目则保持漂移，方差相对平滑。我们通过生成和过滤语义等价的变体，在两个领域（Omni-MATH和SuperGPQA）构建了第一个变体污染基准，并通过微调不同规模和架构的模型（Qwen2.5和Llama3.1）模拟污染。在各种数据集和模型上，DVD在基于困惑度、Min-k%++、编辑距离（CDD）和嵌入相似度的基线中表现一致优于，同时表现出对超参数的强鲁棒性。我们的结果确立了生成分布方差作为检测LLM评估中变体污染的原理性和实用性的指纹。
英文摘要：
Evaluating large language models (LLMs) is increasingly confounded by \emph{variant contamination}: the training corpus contains semantically equivalent yet lexically or syntactically altered versions of test items. Unlike verbatim leakage, these paraphrased or structurally transformed variants evade existing detectors based on sampling consistency or perplexity, thereby inflating benchmark scores via memorization rather than genuine reasoning. We formalize this problem and introduce \textbf{DVD} (\textbf{D}etection via \textbf{V}ariance of generation \textbf{D}istribution), a single-sample detector that models the local output distribution induced by temperature sampling. Our key insight is that contaminated items trigger alternation between a \emph{memory-adherence} state and a \emph{perturbation-drift} state, yielding abnormally high variance in the synthetic difficulty of low-probability tokens; uncontaminated items remain in drift with comparatively smooth variance. We construct the first benchmark for variant contamination across two domains Omni-MATH and SuperGPQA by generating and filtering semantically equivalent variants, and simulate contamination via fine-tuning models of different scales and architectures (Qwen2.5 and Llama3.1). Across datasets and models, \textbf{DVD} consistently outperforms perplexity-based, Min-$k$\%++, edit-distance (CDD), and embedding-similarity baselines, while exhibiting strong robustness to hyperparameters. Our results establish variance of the generation distribution as a principled and practical fingerprint for detecting variant contamination in LLM evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
