如何高效注入知识？预训练大语言模型的知识注入缩放定律
How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
AI总结：
本研究提出知识注入缩放定律，通过分析小模型预测大模型最优领域知识注入量，以平衡专业化与灾难性遗忘，实验验证其有效性和泛化性。
摘要：
大语言模型（LLMs）因其在各类下游任务中展现出的强大通用能力而备受关注。然而，若缺乏针对特定领域的优化，它们在专业知识的基准测试上往往表现不佳，甚至会产生幻觉。近期研究表明，在预训练期间策略性地注入领域知识可以显著提升下游性能。一个关键挑战在于平衡这种注入的权衡：注入过少的领域特定数据会导致专业化不足，而过量注入则会引发对先前习得知识的灾难性遗忘。在本工作中，我们聚焦于过度注入所引发的记忆崩溃现象。通过系统性实验，我们得到两个关键观察：1）临界崩溃点：每个模型都存在一个阈值，超过该阈值后其知识保留能力会急剧下降。2）规模相关性：这些崩溃点与模型规模呈一致的比例关系。基于这些见解，我们提出了一种知识注入缩放定律，通过分析较小规模的模型来预测向大型LLM中注入领域知识的最优数量。跨不同模型规模和预训练token预算的广泛实验验证了我们缩放定律的有效性和泛化性。
英文摘要：
Large language models (LLMs) have attracted significant attention due to their impressive general capabilities across diverse downstream tasks. However, without domain-specific optimization, they often underperform on specialized knowledge benchmarks and even produce hallucination. Recent studies show that strategically infusing domain knowledge during pretraining can substantially improve downstream performance. A critical challenge lies in balancing this infusion trade-off: injecting too little domain-specific data yields insufficient specialization, whereas excessive infusion triggers catastrophic forgetting of previously acquired knowledge. In this work, we focus on the phenomenon of memory collapse induced by over-infusion. Through systematic experiments, we make two key observations, i.e. 1) Critical collapse point: each model exhibits a threshold beyond which its knowledge retention capabilities sharply degrade. 2) Scale correlation: these collapse points scale consistently with the model's size. Building on these insights, we propose a knowledge infusion scaling law that predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. Extensive experiments across different model sizes and pertaining token budgets validate both the effectiveness and generalizability of our scaling law.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
