TiC-LM：一个用于时间持续 LLM 预训练的网络规模基准
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
AI总结：
提出基于 114 次 Common Crawl 转储的 TiC-LM 时间持续预训练基准，并通过时间分层评估发现固定比例重放配合元调度能以 2.6 倍更低计算接近从头训练效果。
摘要：
在历史网络数据上训练的大语言模型（LLM）不可避免地会变得过时。随着新数据出现，我们研究 LLM 的评估策略和更新方法。我们引入一个用于 LLM 时间持续预训练的网络规模数据集，该数据集源自 Common Crawl（CC）的 114 次转储，比以往的持续语言建模基准大数个数量级。我们还在通用 CC 数据和特定领域（Wikipedia、StackExchange 和代码文档）上设计了时间分层评估，以衡量各种持续学习方法在适应新数据的同时保留过往知识的能力。我们的发现表明，在通用 CC 数据上，自回归元调度结合固定比例的旧数据重放，可达到与从头重新训练相当的留出损失，同时所需计算显著更少（2.6 倍）。然而，融合新数据与重放旧数据之间的最佳平衡并不相同：重放对于避免通用网络数据上的遗忘至关重要，但在特定领域则没那么关键。
英文摘要：
Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) - orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
