专家型模型学习更快：预训练中的知识组织塑造知识更新效率
Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency
AI总结：
该研究发现预训练时按时间顺序组织知识的专家型模型，比随机打乱知识训练的通用型模型更新新知识更快，还设计了兼顾历史知识稳定与高效更新的预训练数据组织策略
摘要：
理想的（语言）模型必须持续更新其知识。现有工作主要关注训练后如何使这些更新更高效。我们研究预训练期间随时间变化的知识组织方式如何影响后续更新的效率。在按时间顺序排列的知识上训练的模型会专门针对最近遇到的知识，我们称这类模型为专家型模型（specialists）；相比之下，在相同知识上以随机打乱顺序训练的模型在所有表示的时间段内表现良好，我们称这类模型为通用型模型（generalists）。令人惊讶的是，专家型模型获取新知识所需的更新步骤远少于通用型模型。为解释这一优势，我们从第一性原理推导出一条定律，该定律将更新成本与模型的优化动态相关联，且能紧密预测实验结果。基于这些发现，我们设计了一种预训练数据组织策略，该策略在保留稳定历史知识的同时，能高效更新当前知识。我们还表明，时间知识组织的优势可延伸至现实世界的预训练模型和现有知识更新工具，并能推广到其他知识排序方式。项目页面：this https URL
英文摘要：
An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model's optimization dynamics and that closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings. Project page: https://horizon-lab.org/specialist/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
