一次训练，全部回答：为一次训练成本进行多项预训练实验
Train Once, Answer All: Many Pretraining Experiments for the Cost of One
AI总结：
通过一次训练运行同时进行多项预训练实验，有效降低计算成本并提升研究的严谨性。
摘要：
最近的研究表明，受控的预训练实验是研究训练数据与大语言模型（LLM）行为之间关系的强大工具。然而，预训练的计算成本却成为显著的限制因素。为克服这一限制，我们提出了一种新方法，在单次训练运行中同时进行多个实验。我们通过在2100亿个标记上训练，进行十次实验来验证我们的方法，模型参数达到27亿。尽管模型仅训练一次，我们仍能复现多个先前工作中关于数据污染、中毒和记忆的研究结果。我们还对知识获取、数学推理和水印进行新的研究。例如，我们动态更新训练数据，直到模型获得特定知识。值得注意的是，实验对模型的训练动态和整体性能影响很小。然而，实验之间的相互作用可能在我们的方法中成为混杂因素。我们提出了持续预训练依赖测试（CPDT），一种新的技术来测试持续预训练实验中的相互作用，发现其在我们的设置中可以忽略不计。总体而言，我们的结果表明，在单次训练运行中进行多项预训练实验可以利用计算预算对大模型进行严谨的科学实验。
英文摘要：
Recent work has demonstrated that controlled pretraining experiments are a powerful tool for studying the relationship between training data and large language model (LLM) behavior. However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we propose a new approach where multiple experiments are conducted simultaneously during a single training run. We validate our approach by performing ten experiments while training on 210B tokens, with models of up to 2.7B parameters. Although models are trained only once, we can replicate the results of multiple previous works on data contamination, poisoning, and memorization. We also conduct novel investigations into knowledge acquisition, mathematical reasoning, and watermarking. For example, we dynamically update the training data until a model acquires a particular piece of knowledge. Remarkably, the influence of the experiments on the model's training dynamics and overall performance is minimal. However, interactions between experiments may act as a confounder in our approach. We propose continual pretraining dependence testing (CPDT), a novel technique to test for interactions with continual pretraining experiments, finding them to be negligible in our setup. Overall, our results suggest that performing multiple pretraining experiments within a single training run can enable rigorous scientific experimentation with large models on a compute budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
