通过 LFR 教学法加速大型语言模型预训练：学习、聚焦与复习
Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review
AI总结：
提出 LFR 动态训练范式，通过跟踪学习进度优先复习易遗忘数据，在仅用极少量训练 token 的条件下显著提升 LLM 预训练效率与模型性能。
摘要：
传统大型语言模型（LLM）预训练依赖于自回归语言建模，并从网络规模数据集中随机采样数据。受间隔重复等人类学习技术的启发，我们假设随机采样会导致高训练成本、模型质量下降以及严重的数据遗忘。为了解决这些低效问题，我们提出了 Learn-Focus-Review（LFR）范式——一种适应模型学习进度的动态训练方法。LFR 跟踪模型在数据块（token 序列）上的学习表现，并优先重新访问数据集中更容易被遗忘的困难区域，从而实现更好的记忆和更高效的学习。使用 LFR 范式，我们分别在 SlimPajama 和 OpenWebText 数据集上预训练了 Llama 和 GPT 模型。这些模型在各个领域的下游任务上进行了评估，包括问答、问题解决、常识推理、语言建模和翻译。与在完整数据集上训练的基线模型相比，LFR 持续实现了更低的困惑度和更高的准确率，同时仅使用了 5%--19% 的训练 token。此外，LFR 仅使用 3.2% 的训练 token，就匹配了参数量高达 2 倍的行业标准 Pythia 模型的性能，证明了其有效性和高效性。
英文摘要：
Traditional Large Language Model (LLM) pretraining relies on autoregressive language modeling with randomly sampled data from web-scale datasets. Inspired by human learning techniques like spaced repetition, we hypothesize that random sampling leads to high training costs, lower-quality models, and significant data forgetting. To address these inefficiencies, we propose the Learn-Focus-Review (LFR) paradigm -- a dynamic training approach that adapts to the model's learning progress. LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset that are more prone to being forgotten, enabling better retention and more efficient learning. Using the LFR paradigm, we pretrained Llama and GPT models on the SlimPajama and OpenWebText datasets, respectively. These models were evaluated on downstream tasks across various domains, including question answering, problem-solving, commonsense reasoning, language modeling, and translation. Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy, while using only 5%--19% of the training tokens. Furthermore, LFR matched the performance of industry-standard Pythia models with up to 2$\times$ the parameter count, using just 3.2% of the training tokens, demonstrating its effectiveness and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
