arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.17288cs.AIcs.CL
收藏

ArXiv-to-Model:科学语言模型训练的实践研究

ArXiv-to-Model: A Practical Study of Scientific LM Training

Independent Researcher(独立研究者)

Anuj Gupta

AI总结:

本研究通过从arXiv LaTeX源数据直接训练科学语言模型,探讨了预处理、分词和计算资源对模型训练的影响,提供了工程层面的训练实践与透明分析。

摘要:

尽管前沿大语言模型展现出强大的推理和数学能力,但从原始数据源训练专门领域科学语言模型的实践过程仍缺乏充分的文献记载。在本工作中,我们展示了从覆盖数学、计算机科学和理论物理的arXiv LaTeX原始数据中直接训练一个13.6亿参数的科学语言模型的详细案例研究。我们描述了一个端到端的流程,涵盖元数据过滤、档案验证、LaTeX提取、文本规范化、领域感知的分词以及在受限计算资源(2块A100 GPU)下的密集Transformer训练。通过24次实验运行,我们分析了训练稳定性、扩展行为、数据产出损失和基础设施瓶颈。我们的发现突显了预处理决策如何影响可用的token体积,分词如何影响符号稳定性,以及存储和I/O限制如何可能与计算限制一样成为限制因素。我们进一步分析了收敛动态,并在数据丰富的环境中展示了稳定的训练行为(5200亿预训练token)。与其提出新的架构,本工作提供了一个以工程为基础、透明的从零开始训练小型科学语言模型的过程。我们希望这些见解能支持那些在中等计算预算下寻求构建领域专用模型的研究人员。

英文摘要:

While frontier large language models demonstrate strong reasoning and mathematical capabilities, the practical process of training domain-specialized scientific language models from raw sources remains under-documented. In this work, we present a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics. We describe an end-to-end pipeline covering metadata filtering, archive validation, LaTeX extraction, text normalization, domain-aware tokenization, and dense transformer training under constrained compute (2xA100 GPUs). Through 24 experimental runs, we analyze training stability, scaling behavior, data yield losses, and infrastructure bottlenecks. Our findings highlight how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors. We further analyze convergence dynamics and show stable training behavior in a data-rich regime (52B pretraining tokens). Rather than proposing a novel architecture, this work provides an engineering-grounded, transparent account of training a small scientific language model from scratch. We hope these insights support researchers operating under moderate compute budgets who seek to build domain-specialized models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑