面向成员推理攻击的基于采样的伪似然方法
Sampling-based Pseudo-Likelihood for Membership Inference Attacks
AI总结:
针对现有基于似然的成员推理攻击无法应用于无似然输出的专有大语言模型的问题,提出基于采样伪似然的SaMIA方法,仅用模型生成文本的n-gram匹配度判定成员关系,性能与现有似然类方法相当。
摘要:
大语言模型(LLM)基于大规模网络数据训练,这使得人们难以掌握每段文本的训练贡献,进而带来训练数据中基准数据、个人信息、受版权保护文本等敏感数据泄露的风险。成员推理攻击(MIA)用于判定给定文本是否被包含在模型训练数据中,目前正受到广泛关注。此前的MIA研究表明,基于似然的分类方法能有效检测LLM的数据泄露问题。但现有方法无法应用于ChatGPT、Claude 3等部分专有模型,因为用户无法获取这类模型的似然值。本研究提出了用于成员推理攻击的基于采样的伪似然(SPL)方法(SaMIA),该方法仅利用LLM生成的文本计算SPL以检测数据泄露。SaMIA将目标文本作为参考文本,将LLM的多次输出作为文本样本,计算n-gram匹配程度作为SPL,进而判定文本是否属于训练数据。即使无法获取似然值,SaMIA的性能也与现有基于似然的方法相当。
英文摘要:
Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data such as benchmarks, personal information, and copyrighted texts in the training data. Membership Inference Attacks (MIA), which determine whether a given text is included in the model's training data, have been attracting attention. Previous studies of MIAs revealed that likelihood-based classification is effective for detecting leaks in LLMs. However, the existing methods cannot be applied to some proprietary models like ChatGPT or Claude 3 because the likelihood is unavailable to the user. In this study, we propose a Sampling-based Pseudo-Likelihood (\textbf{SPL}) method for MIA (\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks. The SaMIA treats the target text as the reference text and multiple outputs from the LLM as text samples, calculates the degree of $n$-gram match as SPL, and determines the membership of the text in the training data. Even without likelihoods, SaMIA performed on par with existing likelihood-based methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。