扩展即时语言模型

中文摘要 AI 辅助

在无限制的互联网语料库上训练的大型语言模型不可避免地嵌入了来自未来的信息，引入了前瞻性偏差，这损害了金融和社会科学中回测和因果推断的有效性。即时语言模型通过仅在每个日历日期之前可用的文本上进行训练来消除这种泄漏，但现有努力通常产生的模型在性能上大大落后于无约束的模型。我们表明，通过扩展规模可以大幅缩小这种性能差距。我们在来自FineWeb的1万亿个按时间顺序过滤的令牌上训练了多达40亿参数的仅解码器变压器，构建了一系列跨越2013 - 2024年的月度模型检查点。在一系列常识推理和语言理解基准测试中，我们的模型接近了在不受时间限制的数据上训练的可比规模的领先开放权重模型（如Gemma - 3 - 4B和LLaMA - 7B）的性能，不过在一些任务上仍存在性能差距。通过LoRA进行指令微调进一步提高了下游可用性。我们发布了完整的管道，包括数据集构建、训练基础设施和评估代码，以实现可重复的即时语言建模并支持需要严格时间有效性的研究应用。

英文摘要

Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models--trained exclusively on text available up to each calendar date--eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline--including dataset construction, training infrastructure, and evaluation code--to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

发表机构

• Yale School of Management（耶鲁大学管理学院）
• AQR Capital Management（AQR资产管理公司）
• NBER（美国国家经济研究局）
• Swiss Finance Institute（瑞士金融研究所）
• EPFL（洛桑联邦理工学院）
• CEPR（经济政策研究中心）

机构由 AI 辅助整理，请以论文原文为准。