arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.17817cs.CLcs.LG
收藏

揭秘大型语言模型中的逐字记忆

Demystifying Verbatim Memorization in Large Language Models

Stanford University(斯坦福大学)

Jing Huang, Diyi Yang, Christopher Potts

AI总结:

本研究开发受控框架探究LLM逐字记忆机制,发现其与模型通用能力深度交织,现有遗忘方法难以在不损害模型性能的前提下有效消除该现象。

摘要:

大型语言模型(LLMs)经常逐字记忆长序列,这往往带来严重的法律和隐私影响。许多先前的工作使用观测数据研究了这种逐字记忆。为了补充这些工作,我们开发了一个框架,通过从 Pythia 检查点继续预训练并注入序列,在受控环境中研究逐字记忆。我们发现:(1)非微量的重复是发生逐字记忆的必要条件;(2)较晚的(推测更好的)检查点更可能逐字记忆序列,即使对于分布外序列也是如此;(3)记忆序列的生成由编码高级特征的分布式模型状态触发,并重要地利用了通用语言建模能力。在这些见解的指导下,我们开发了压力测试来评估遗忘方法,发现它们通常无法移除逐字记忆的信息,同时还会降低 LM 的性能。总体而言,这些发现挑战了逐字记忆源于特定模型权重或机制的假设。相反,逐字记忆与 LM 的通用能力交织在一起,因此在不降低模型质量的情况下将很难隔离和抑制。

英文摘要:

Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑