新数据如何渗透LLM知识及如何稀释它
How new data permeates LLM knowledge and how to dilute it
AI总结：
针对LLM学习新信息时出现的“启动”效应（新事实在无关语境中被不当应用），本文构建Outlandish数据集探究该现象，提出两种技术将不良启动效应降低50-95%，为LLM知识更新机制提供见解并提升知识插入特异性。
摘要：
大型语言模型通过基于梯度的更新积累进行学习和持续学习，但新信息片段如何影响现有知识，从而产生有益泛化与问题幻觉，仍知之甚少。我们证明，学习新信息时LLM会表现出“启动”效应：学习新事实会导致模型在无关语境中不当应用该知识。为系统研究此现象，我们引入“Outlandish”——一个精心构建的含1320个多样文本样本的数据集，用于探究新知识如何渗透LLM现有知识库。通过该数据集，我们发现学习新信息后的启动程度可通过测量学习前关键词的token概率来预测，此关系在不同模型架构（PALM-2、Gemma、Llama）、规模及训练阶段均稳健成立。最后，我们开发两种新方法调节新知识对现有模型行为的影响：（1）“垫脚石”文本增强策略；（2）“ignore-k”更新剪枝法。这些方法将不良启动效应降低50-95%，同时保留模型学习新信息的能力。我们的发现既为LLM学习机制提供实证见解，也为提升语言模型知识插入的特异性提供实用工具。更多材料：https://sunchipsster1.github.io/projects/outlandish/
英文摘要：
Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts. To systematically study this phenomenon, we introduce "Outlandish," a carefully curated dataset of 1320 diverse text samples designed to probe how new knowledge permeates through an LLM's existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) a ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update pruning method. These approaches reduce undesirable priming effects by 50-95\% while preserving the model's ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: https://sunchipsster1.github.io/projects/outlandish/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
