We propose the Data Contamination Quiz (DCQ), a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifically, we frame data contamination detection as a series of multiple-choice questions, devising a quiz format wherein three perturbed versions of each instance, subsampled from a specific dataset partition, are created. These changes only include word-level perturbations. The generated perturbations, along with the original dataset instance, form the options in the DCQ, with an extra option accommodating the selection of none of the provided options. Given that the only distinguishing signal among the options is the exact wording with respect to the original dataset instance, an LLM, when tasked with identifying the original dataset instance, gravitates towards selecting the original one if it has been exposed to it. While accounting for positional biases in LLMs, the quiz performance reveals the contamination level for the tested model with the dataset partition to which the quiz pertains. Applied to various datasets and LLMs, under controlled and uncontrolled contamination, our findings, while fully lacking access to training data and model parameters, suggest that DCQ achieves state-of-the-art results and uncovers greater contamination levels through memorization compared to existing methods. Also, it proficiently bypasses more safety filters, especially those set to avoid generating copyrighted content.
LogProber: Disentangling confidence from contamination in LLM responses
LNC2, INSERM, Paris, France(LNC2,法国国家医学研究院,巴黎);DEC, ENS, PSL, Paris, France(DEC,巴黎高等师范学院,巴黎);Flowers AI & CogSci Lab Centre Inria de L’Université de Bordeaux(AI与认知科学实验室,Inria-波尔多大学研究中心)
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Key Laboratory of High Confidence Software Technologies (Peking University)(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);Beijing Institute of Control Engineering(北京控制工程研究所);China Academy of Space Technology(中国空间技术研究院)
Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets
Guanghua School of Management, Peking University(北京大学光华管理学院);College of Artificial Intelligence, Jilin University(吉林大学人工智能学院);School of Mathematical Sciences, Peking University(北京大学数学科学学院)
DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation
Beihang University(北京航空航天大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学)
LNE-Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models
Zhejiang University(浙江大学);FaceMind Corporation(FaceMind公司);University of Zurich(苏黎世大学);The Chinese University of Hong Kong(香港中文大学);Westlake University(西湖大学)
A Survey on Data Contamination for Large Language Models
College of Software, Jilin University(吉林大学软件学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)