从潜在思维中学习的推理
Reasoning to Learn from Latent Thoughts
AI总结：
针对预训练数据瓶颈，本文提出显式推断文本背后的潜在思维以增强数学预训练，并用 EM 让 1B LM 无强教师自举迭代，显著提升数据效率。
摘要：
语言模型（LM）预训练的计算规模扩张速度已经超过人类撰写文本的增长速度，这引发了数据将成为 LM 扩张瓶颈的担忧。为了在这种数据受限的条件下继续扩展预训练，我们提出，显式建模并推断构成文本生成过程基础的“潜在思维”，可以显著提升预训练数据效率。直观而言，我们的方法将网络文本视为冗长的人类思维过程经过压缩后的最终产物，而潜在思维包含对数据高效学习至关重要的重要上下文知识与推理步骤。我们通过面向数学的数据受限持续预训练，实验证明了该方法的有效性。首先，我们表明，用于推断潜在思维的合成数据方法相较使用同等数量原始数据进行训练，能够显著提升数据效率。此外，我们展示了在没有强教师模型情况下的潜在思维推断：LM 使用 EM 算法迭代提升被训练 LM 的能力以及经过思维增强的预训练数据质量，从而“自举提升自身性能”。我们表明，一个 1B LM 可以在至少三次迭代中自举提升性能，并显著优于在原始数据上训练的基线；在执行 E 步时，增加推理计算会带来不断增大的收益。推理扩展与 EM 迭代带来的收益，为扩展数据受限预训练揭示了新的机遇。
英文摘要：
Compute scaling for language model (LM) pretraining has outpaced the growth of human-written texts, leading to concerns that data will become the bottleneck to LM scaling. To continue scaling pretraining in this data-constrained regime, we propose that explicitly modeling and inferring the \emph{latent thoughts} that underlie the text generation process can significantly improve pretraining data efficiency. Intuitively, our approach views web text as the compressed final outcome of a verbose human thought process and that the latent thoughts contain important contextual knowledge and reasoning steps that are critical to data-efficient learning. We empirically demonstrate the effectiveness of our approach through data-constrained continued pretraining for math. We first show that synthetic data approaches to inferring latent thoughts significantly improve data efficiency over training on the same amount of raw data. Furthermore, we demonstrate latent thought inference without a strong teacher, where an LM \emph{bootstraps its own performance} by using an EM algorithm to iteratively improve the capability of the trained LM and the quality of thought-augmented pretraining data. We show that a 1B LM can bootstrap its performance across at least three iterations and significantly outperform baselines trained on raw data, with increasing gains from additional inference compute when performing the E-step. The gains from inference scaling and EM iterations suggest new opportunities for scaling data-constrained pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
