Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA.
Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directions
Hokkaido University(北海道大学);China University of Mining and Technology(中国矿业大学);Institute of Science Tokyo(东京科学大学);Hong Kong University of Science and Technology(香港科技大学)
ICAS: Detecting Training Data from Autoregressive Image Generative Models
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学深圳学院);Shenzhen ShenNong Information Technology Co., Ltd.(深圳深农信息技术有限公司)
Membership Inference for Contrastive Pre-training Models with Text-only PII Queries
Beijing Electronic Science and Technology Institute(北京电子科技研究所);Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信人工智能研究院(TeleAI));Nanyang Technological University(南洋理工大学);Nanjing University(南京大学);School of Computing and Information Technology, Great Bay University(广东大湾区大学计算机与信息科技学院)
Textual Unlearning Gives a False Sense of Unlearning
The State Key Laboratory of Blockchain and Data Security, Zhejiang University, P. R. China(区块链与数据安全国家重点实验室,浙江大学,中华人民共和国);School of Cyber Science and Technology, Zhejiang University, P. R. China(网络安全与技术学院,浙江大学,中华人民共和国);ETH Zurich, Switzerland(苏黎世联邦理工学院,瑞士)