arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.05968cs.CR
收藏

Nob-MIAs:基于事后数据集构建的大语言模型无偏成员推断攻击评估

Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction

Cédric Eichler, Nathan Champeil, Nicolas Anciaux, Alexandra Bensamoun, Heber Hwang Arcolezi, José Maria De Fuentes

AI总结:

针对大语言模型成员推断攻击评估受分布偏差干扰的问题,本文提出事后假设下的无偏数据集构建方法,实验验证该方法可实现公平评估,且多数MIA在去偏后性能接近随机。

摘要:

大语言模型(LLMs)的兴起引发了法律和伦理层面的担忧,尤其是其训练数据集中未经授权使用受版权保护材料的问题,这已导致多家被指控未经许可使用受保护内容的科技公司遭到起诉。成员推断攻击(MIAs)旨在检测特定文档是否被用于给定LLM的预训练,但这类攻击的有效性会受到时间偏移、n-gram重叠等偏差的削弱。\n本文在事后假设(即承认成员与非成员数据集之间存在固有分布偏差)的框架下,针对训练集可部分推断的LLM,开展MIA评估研究。我们提出并验证了用于构建“无偏”且“不可分类”数据集的算法,以实现更公平的MIA评估。在Gutenberg数据集上针对OpenLlama和Pythia开展的实验表明,仅中和已知偏差是不够的。我们的方法生成的无偏事后数据集,其AUC-ROC得分与此前在真正随机数据集上得到的结果相当,验证了我们方法的有效性。总体而言,MIA的结果接近随机水平,仅有一种攻击在随机数据集和我们构建的数据集上均有效,但在消除偏差后其性能会下降。

英文摘要:

The rise of Large Language Models (LLMs) has triggered legal and ethical concerns, especially regarding the unauthorized use of copyrighted materials in their training datasets. This has led to lawsuits against tech companies accused of using protected content without permission. Membership Inference Attacks (MIAs) aim to detect whether specific documents were used in a given LLM pretraining, but their effectiveness is undermined by biases such as time-shifts and n-gram overlaps. This paper addresses the evaluation of MIAs on LLMs with partially inferable training sets, under the ex-post hypothesis, which acknowledges inherent distributional biases between members and non-members datasets. We propose and validate algorithms to create ``non-biased'' and ``non-classifiable'' datasets for fairer MIA assessment. Experiments using the Gutenberg dataset on OpenLamma and Pythia show that neutralizing known biases alone is insufficient. Our methods produce non-biased ex-post datasets with AUC-ROC scores comparable to those previously obtained on genuinely random datasets, validating our approach. Globally, MIAs yield results close to random, with only one being effective on both random and our datasets, but its performance decreases when bias is removed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑