arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.06233cs.CLcs.AIcs.LG
收藏

数据污染测验:一种检测和估计大型语言模型数据污染的工具

Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models

Department of Computer Science, University of Arizona(计算机科学系,亚利桑那大学)

Shahriar Golchin, Mihai Surdeanu

AI总结:

提出 Data Contamination Quiz (DCQ),将数据污染检测转化为多选题测验,通过词级扰动选项利用模型记忆揭示污染程度,在无需访问训练数据和模型参数下达到 SOTA 并有效绕过安全过滤器。

摘要:

我们提出了 Data Contamination Quiz (DCQ),一种简单有效的方法,用于检测大型语言模型 (LLMs) 中的数据污染并估计其污染程度。具体而言,我们将数据污染检测构建为一系列多选题,设计了一种测验格式,其中为从特定数据集分区中下采样的每个实例创建三个扰动版本。这些更改仅包括词级扰动。生成的扰动与原始数据集实例一起构成 DCQ 中的选项,并设有一个额外选项以供选择不提供任何选项。鉴于各选项之间唯一的区分信号是与原始数据集实例完全一致的措辞,当 LLM 被要求识别原始数据集实例时,如果它曾接触过该实例,就会倾向于选择原始实例。在考虑 LLM 中位置偏差的同时,测验性能揭示了测试模型在测验所涉数据集分区上的污染水平。在受控和非受控污染下应用于各种数据集和 LLMs,我们的研究结果表明,在完全无法访问训练数据和模型参数的情况下,DCQ 达到了最先进的结果,并通过记忆揭示了比现有方法更高的污染水平。此外,它还能熟练地绕过更多安全过滤器,尤其是那些旨在避免生成受版权保护内容的安全过滤器。

英文摘要:

We propose the Data Contamination Quiz (DCQ), a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifically, we frame data contamination detection as a series of multiple-choice questions, devising a quiz format wherein three perturbed versions of each instance, subsampled from a specific dataset partition, are created. These changes only include word-level perturbations. The generated perturbations, along with the original dataset instance, form the options in the DCQ, with an extra option accommodating the selection of none of the provided options. Given that the only distinguishing signal among the options is the exact wording with respect to the original dataset instance, an LLM, when tasked with identifying the original dataset instance, gravitates towards selecting the original one if it has been exposed to it. While accounting for positional biases in LLMs, the quiz performance reveals the contamination level for the tested model with the dataset partition to which the quiz pertains. Applied to various datasets and LLMs, under controlled and uncontrolled contamination, our findings, while fully lacking access to training data and model parameters, suggest that DCQ achieves state-of-the-art results and uncovers greater contamination levels through memorization compared to existing methods. Also, it proficiently bypasses more safety filters, especially those set to avoid generating copyrighted content.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑