arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09363cs.CLcs.CR
收藏

大型语言模型的版权陷阱

Copyright Traps for Large Language Models

Imperial College London(帝国理工学院);CentraleSupélec(中央高等电力学院);Université Paris-Saclay(巴黎萨克雷大学)

Matthieu Meeus, Igor Shilov, Manuel Faysse, Yves-Alexandre de Montjoye

AI总结:

本文提出“版权陷阱”方法,通过在原始内容中插入虚构条目,有效检测不自然记忆的中型LLMs是否使用了受版权保护的材料,并揭示了记忆与序列重复及困惑度间的因果关系。

摘要:

使用受版权保护的内容来训练大型语言模型(LLMs)的合理使用问题正在被积极讨论。文档级推理已被提出作为一项新任务:通过对训练模型的黑盒访问,推断某段内容是否在训练期间被见过。然而,现有最先进的方法依赖于内容(部分)的自然发生记忆。虽然这些方法对显著记忆的模型非常有效,但我们假设——并随后证实——它们对不会自然记忆的模型(例如10亿参数的中型模型)不起作用。我们在此提出使用版权陷阱,即在原始内容中包含虚构条目,以检测LLMs中受版权保护材料的使用,重点关注不会自然发生记忆的模型。我们精心设计了一个随机对照实验设置,将陷阱插入原始内容(书籍)中,并从头开始训练一个1.3B参数的LLM。我们首先验证,使用现有方法无法检测到目标模型中的内容使用。然后我们表明,与直觉相反,即使重复显著次数(100次)的中等长度陷阱句子也无法使用现有方法检测到。然而,我们证明,重复大量次数的较长序列可以被可靠地检测到(AUC=0.75),并可用作版权陷阱。除了版权应用外,我们的发现还促进了对LLM记忆的研究:随机对照设置使我们能够得出记忆与某些序列属性(如模型训练数据中的重复和困惑度)之间的因果关系。

英文摘要:

Questions of fair use of copyright-protected content to train Large Language Models (LLMs) are being actively debated. Document-level inference has been proposed as a new task: inferring from black-box access to the trained model whether a piece of content has been seen during training. SOTA methods however rely on naturally occurring memorization of (part of) the content. While very effective against models that memorize significantly, we hypothesize--and later confirm--that they will not work against models that do not naturally memorize, e.g. medium-size 1B models. We here propose to use copyright traps, the inclusion of fictitious entries in original content, to detect the use of copyrighted materials in LLMs with a focus on models where memorization does not naturally occur. We carefully design a randomized controlled experimental setup, inserting traps into original content (books) and train a 1.3B LLM from scratch. We first validate that the use of content in our target model would be undetectable using existing methods. We then show, contrary to intuition, that even medium-length trap sentences repeated a significant number of times (100) are not detectable using existing methods. However, we show that longer sequences repeated a large number of times can be reliably detected (AUC=0.75) and used as copyright traps. Beyond copyright applications, our findings contribute to the study of LLM memorization: the randomized controlled setup enables us to draw causal relationships between memorization and certain sequence properties such as repetition in model training data and perplexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑