arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.06443cs.AI
收藏

暴露隐私缺口:针对LLM对齐偏好数据的成员推理攻击

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

Amazon Inc.(亚马逊公司)

Qizhang Feng, Siva Rajesh Kasa, Santhosh Kumar Kasa, Hyokun Yun, Choon Hui Teo, Sravan Babu Bodapati

AI总结:

本文研究了LLM对齐中偏好数据的隐私漏洞,理论上论证了DPO模型比PPO模型更易受成员推理攻击,并提出专门攻击框架PREMIA进行实证验证。

摘要:

大型语言模型(LLMs)因其卓越的自然语言能力而得到广泛应用。然而,在真实场景中部署时,将LLM对齐以符合人类可接受标准至关重要。近端策略优化(PPO)和直接偏好优化(DPO)等方法利用人类偏好数据在优化LLM方面取得了显著进展。然而,利用此类偏好数据所固有的隐私问题尚未得到充分研究。本文研究了使用DPO和PPO这两种广泛使用方法对齐的LLM对成员推理攻击(MIA)的脆弱性。本研究有两项主要贡献:首先,我们从理论上论证了DPO模型比PPO模型更容易受到MIA攻击;其次,我们引入了一个专门用于分析偏好数据的基于参考的攻击框架PREMIA(偏好数据MIA)。使用PREMIA和现有基线,我们实证表明DPO模型对MIA具有相对更高的脆弱性。

英文摘要:

Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑