Membership Inference Attacks (MIAs) have recently been employed to determine whether a specific text was part of the pre-training data of Large Language Models (LLMs). However, existing methods often misinfer non-members as members, leading to a high false positive rate, or depend on additional reference models for probability calibration, which limits their practicality. To overcome these challenges, we introduce a novel framework called Automatic Calibration Membership Inference Attack (ACMIA), which utilizes a tunable temperature to calibrate output probabilities effectively. This approach is inspired by our theoretical insights into maximum likelihood estimation during the pre-training of LLMs. We introduce ACMIA in three configurations designed to accommodate different levels of model access and increase the probability gap between members and non-members, improving the reliability and robustness of membership inference. Extensive experiments on various open-source LLMs demonstrate that our proposed attack is highly effective, robust, and generalizable, surpassing state-of-the-art baselines across three widely used benchmarks. Our code is available at: \href{https://github.com/Salehzz/ACMIA}{\textcolor{blue}{Github}}.
DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation
School of Cyber Science and Technology, Shandong University(山东大学计算机科学与技术学院);Department of Engineering Software, School of Civil Engineering, Shandong University(山东大学土木工程学院工程软件系);State Key Laboratory of Cryptography and Digital Economy Security, Shandong University(山东大学密码与数字经济安全国家重点实验室);Shandong Key Laboratory of Artificial Intelligence Security, Shandong University(山东省人工智能安全重点实验室)
On the Impact of Uncertainty and Calibration on Likelihood-Ratio Membership Inference Attacks
Beijing Key Laboratory of Network System Architecture and Convergence, School of Information and Communication Engineering, Beijing University of Posts and Telecommunications(北京网络系统架构与融合重点实验室,信息与通信工程学院,北京邮电大学);King’s Communications, Learning & Information Processing (KCLIP) lab, Department of Engineering, King’s College London(国王学院通讯、学习与信息处理(KCLIP)实验室,工程学院,伦敦国王学院)
GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心);University of Maryland, College Park(马里兰大学帕克分校);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
Automatic re-calibration of quantum devices by reinforcement learning
Computer Vision Center (CVC)(计算机视觉中心);Instituto de Física La Plata (IFLP)(拉普拉塔物理研究所);CONICET(阿根廷科学与技术研究委员会);Universidad Nacional de La Plata (UNLP)(拉普拉塔国立大学);Departamento de Ciencias Básicas, Facultad de Ingeniería, Universidad Nacional de La Plata (UNLP)(拉普拉塔国立大学工程学院基础科学系);Department of Computer Science, Universitat Autònoma de Barcelona (UAB)(巴塞罗那自治大学计算机科学系)
Improving Membership Inference in ASR Model Auditing with Perturbed Loss Features
INESC-ID/Técnico, University of Lisbon(里斯本大学高等理工学院INESC-ID研究所);Technical University Munich(慕尼黑工业大学);Fraunhofer AISEC(弗劳恩霍夫应用信息安全研究所);Carnegie Mellon University(卡内基梅隆大学)
本文提出结合高斯与对抗扰动的损失特征用于 ASR 模型成员推断,显著提升样本级性能,并为模型审计提供新思路。