通过困惑度估计污染：量化语言模型评估中的记忆现象
Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation
AI总结：
针对训练数据保密导致难以审计基准污染的问题，本文提出仅用困惑度量化语言模型记忆程度的方法，并发现阅读理解和摘要基准污染显著，多项选择题污染较轻。
摘要：
由于大型语言模型的海量训练语料库往往会在无意中包含基准样本，模型评估中的数据污染正变得日益普遍。因此，污染分析已成为可靠模型评估中不可避免的一环。然而，现有的污染分析方法需要访问完整训练数据，而近期模型的训练数据通常保密。这使研究界无法严格审计这些模型，也无法准确评估其能力。本文提出一种无需访问完整训练集即可量化污染的新方法，通过困惑度衡量污染程度。分析表明，近期基础模型在流行的阅读理解和摘要基准上存在显著记忆，而多项选择题受到的污染似乎较轻。
英文摘要：
Data contamination in model evaluation is getting increasingly prevalent as the massive training corpora of large language models often unintentionally include benchmark samples. Therefore, contamination analysis has became an inevitable part of reliable model evaluation. However, existing method of contamination analysis requires the access of the entire training data which is often confidential for recent models. This prevent the community to rigorously audit these models and conduct accurate assessment of their capability. In this paper, we propose a novel method to quantify contamination without the access of the full training set, that measure the extent of contamination with perplexity. Our analysis provides evidence of significant memorisation of recent foundation models in popular reading comprehension, summarisation benchmarks, while multiple choice appears less contaminated.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
