理解数据时间性对大型语言模型预训练的影响
Understanding Data Temporality Impact on Large Language Models Pre-training
AI总结：
研究预训练数据顺序对大型语言模型获取时间敏感事实知识的影响，通过构建包含7000多个时间相关问题的基准并训练60亿参数模型，发现按时间顺序训练比随机打乱训练能产生更及时和精确的知识。
摘要：
大型语言模型（LLMs）通常在打乱顺序的语料库上进行训练，导致模型的知识在训练时被冻结，其时间基础仍然难以理解。在这项工作中，我们研究了预训练动态对获取时间敏感事实知识的影响，特别关注数据顺序。我们的主要贡献有两方面。首先，我们引入了一个包含7000多个时间基础问题的综合基准和一个评估协议，能够分析模型是否将事实与其对应的时间段正确关联。其次，我们在按时间顺序排列的Common Crawl快照上预训练了60亿参数的模型，并将其与标准的随机打乱预训练进行比较。我们的结果表明，按顺序训练的模型在通用语言理解和常识方面与随机打乱的基线相当，同时始终表现出更及时和精确的时间知识。按时间顺序的预训练提高了事实的新鲜度，而随机打乱的预训练在较旧的数据上表现更好，可能是由于事实重复增加。这些发现，连同我们在https://github.com/kyutai-labs/kairos 发布的代码、在https://huggingface.co/collections/kyutai/kairos 发布的检查点和数据集，为LLMs的持续学习未来研究提供了基础。
英文摘要：
Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
