LLM基准数据集应具备抗污染性
LLM Benchmark Datasets Should Be Contamination-Resistant
AI总结：
本文探讨了LLM基准数据集应具备抗污染性，提出通过改进数据集设计和架构来提高其可靠性和通用性。
摘要：
基准数据集对于可重复、可靠和具有判别性的LLM评估至关重要。然而，最近的研究表明，许多基准数据集包含在预训练语料库中，即被污染，这降低了它们作为可靠模型泛化度量的价值。在本文中，我们主张基准数据集应具备抗污染性，即不可学习但支持推理。为此，我们首先强调基准数据集污染的广泛存在，并概述抗污染数据集的性质。其次，我们强调Transformer架构中推理和训练流程之间的不对称性可以用来支持抗污染性。第三，我们概述了使这些数据集在各种LLM架构之间互操作的数学进展。基于上述内容，我们呼吁社区通过：(i) 推动新的抗污染方法，(ii) 开发支持方法和平台，以及(iii) 在现有评估流程中采用抗污染基准来确保LLM评估的可靠性。
英文摘要：
Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark datasets are included in pretraining corpora, i.e., $\textit{contaminated}$, which diminishes their value as reliable measures of model generalization. In this paper, we argue that benchmark datasets should be $\textit{contamination-resistant}$, i.e., $\textit{unlearnable}$, but support $\textit{inference}$. To accomplish this, we first highlight the wide prevalence of benchmark dataset contamination and outline the properties of contamination-resistant datasets. Second, we highlight how the asymmetry between the inference and training pipelines in the Transformer architecture can be leveraged to support contamination-resistance. Third, we outline mathematical advancements to make these datasets interoperable across various LLM architectures. Based on the above, we call on the community to ensure the reliability of LLM benchmarking by: (i) advancing novel contamination-resistant methodologies, (ii) developing supporting methods and platforms, and (iii) adopting contamination-resistant benchmarks into existing evaluation pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
