程序化预训练:用抽象数据预热语言模型
Procedural Pretraining: Warming Up Language Models with Abstract Data
AI总结:
提出程序化预训练方法,通过在抽象结构化数据(如形式语言生成的程序数据)上预训练语言模型,显著提升其推理能力并加速后续语义知识学习,实验表明仅需0.1-0.3%的程序数据即可超越标准预训练。
摘要:
直接在网络规模语料库上预训练语言模型是当前的主流范式。我们研究了一种替代方案:首先让模型接触抽象结构化数据,以简化后续丰富语义知识的获取,类似于人类在学习高级推理之前先学习简单逻辑和数学。我们关注由形式语言和其他简单算法生成的程序数据作为此类抽象数据。首先,我们诊断了不同形式的程序数据能够提升的算法技能,通常效果显著。例如,当模型在Dyck序列(平衡括号)上预训练时,上下文召回(大海捞针)的准确率从10%跃升至98%。其次,我们研究了这些增益如何反映在更大模型(高达1.3B参数)的预训练中。我们发现,仅在前端加入0.1%至0.3%的程序数据,就能显著优于在自然语言、代码和非正式数学(C4、CodeParrot和DeepMind-Math数据集)上的标准预训练。值得注意的是,这也使得模型仅需原始数据的55/67/86%即可达到相同的损失值,从而相应地减少FLOPs。第三,我们探索了这些收益背后的机制,发现程序化预训练在注意力层和MLP层中都注入了非平凡的结构。前者对于结构化领域(如代码)尤为重要,后者对于语言领域重要。最后,我们为组合多种形式的程序数据铺平了道路。我们的结果表明,程序化预训练是一种简单、轻量级的方法,能够提升性能并加速语言模型预训练,最终揭示了在LLM中将知识获取与推理分离的前景。
英文摘要:
Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease the subsequent acquisition of rich semantic knowledge, much like humans learning simple logic and mathematics before higher reasoning. We focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, the accuracy of context recall (Needle-in-a-haystack) jumps from 10 to 98% when a model is pretrained on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1 to 0.3% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this also enables the models to reach the same loss value with only 55/67/86% of the original data and thus a comparable reduction in FLOPs. Third, we explore the mechanisms behind the benefits and find that procedural pretraining instills non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means of improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。