数据污染可跨越语言壁垒
Data Contamination Can Cross Language Barriers
AI总结：
该研究发现了能规避现有检测的跨语言大语言模型基准污染形式，提出基于泛化的检测方法并验证其有效性，同时探讨了该污染形式的潜在应用价值。
摘要：
大语言模型（LLM）开发过程的不透明性，正引发人们对预训练数据中公共基准可能被污染的日益担忧。现有污染检测方法通常基于训练数据与评估数据的文本重叠度，这种方法可能过于表面，无法反映更深层次的污染形式。\n本文首先提出了一种跨语言形式的污染，它会虚高LLM的性能，同时能规避当前的检测方法——这种污染是通过让LLM在基准测试集的翻译版本上过拟合而刻意注入的。随后，我们提出了基于泛化的方法来揭露这种隐藏极深的污染。具体而言，我们通过将基准测试原题的错误选项替换为其他题目的正确选项来修改原始基准，再考察LLM的性能变化。受污染的模型几乎无法泛化到这种更简单的场景中，此时错误选项甚至“连错都算不上”，因为在模型的记忆中所有选项都是正确的。\n实验结果表明，跨语言污染可以轻易骗过现有检测方法，但逃不过我们的方法。此外，我们还讨论了跨语言污染在解读LLM工作机制、以及在对LLM进行后训练以提升多语言能力方面的潜在应用价值。我们使用的代码和数据集可从https://github.com/ShangDataLab/Deep-Contam获取。
英文摘要：
The opacity in developing large language models (LLMs) is raising growing concerns about the potential contamination of public benchmarks in the pre-training data. Existing contamination detection methods are typically based on the text overlap between training and evaluation data, which can be too superficial to reflect deeper forms of contamination. In this paper, we first present a cross-lingual form of contamination that inflates LLMs' performance while evading current detection methods, deliberately injected by overfitting LLMs on the translated versions of benchmark test sets. Then, we propose generalization-based approaches to unmask such deeply concealed contamination. Specifically, we examine the LLM's performance change after modifying the original benchmark by replacing the false answer choices with correct ones from other questions. Contaminated models can hardly generalize to such easier situations, where the false choices can be \emph{not even wrong}, as all choices are correct in their memorization. Experimental results demonstrate that cross-lingual contamination can easily fool existing detection methods, but not ours. In addition, we discuss the potential utilization of cross-lingual contamination in interpreting LLMs' working mechanisms and in post-training LLMs for enhanced multilingual capabilities. The code and dataset we use can be obtained from \url{https://github.com/ShangDataLab/Deep-Contam}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
