As large language models (LLMs) are trained on increasingly vast and opaque text corpora, determining which data contributed to training has become essential for copyright enforcement, compliance auditing, and user trust. While prior work focuses on detecting whether a dataset was used in training (membership inference), the complementary problem -- verifying that a dataset was not used -- has received little attention. We address this gap by introducing PRISM, a test that detects dataset-level non-membership using only grey-box access to model logits. Our key insight is that two models that have not seen a dataset exhibit higher rank correlation in their normalized token log probabilities than when one model has been trained on that data. Using this observation, we construct a correlation-based test that detects non-membership. Empirically, PRISM reliably rules out membership in training data across all datasets tested while avoiding false positives, thus offering a framework for verifying that specific datasets were excluded from LLM training.
Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding
National University of Singapore(新加坡国立大学);University of California, Merced(加州大学默塞德分校);University of California, Los Angeles(加州大学洛杉矶分校);University of Queensland(昆士兰大学)
Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models
Parameter Lab(参数实验室);Ubiquitous Knowledge Processing Lab(普适知识处理实验室);Technical University of Darmstadt(达姆施塔特工业大学);NAVER AI Lab(NAVER人工智能实验室);University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心)
Automated Detection of Pre-training Text in Black-box LLMs
Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, China(可信分布式计算与服务重点实验室,北京邮电大学,中国);Zhongguancun Laboratory, China(中关村实验室,中国)
Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(中国科学技术大学认知智能全国重点实验室);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd(科大讯飞股份有限公司(华中)讯飞人工智能研究院)
Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document
Cornell University(康奈尔大学);Sofia University(索非亚大学);Qatar Computing Research Institute, HBKU(哈马德·本·哈利法大学卡塔尔计算研究所);University of Padova(帕多瓦大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)