模态,一种用于大规模大语言模型训练和研究的PyTorch原生框架
Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research
AI总结:
Modalities是一个基于PyTorch的框架,旨在通过整合先进的并行策略和模块化设计,提升大规模大语言模型训练和研究的效率与可扩展性。
摘要:
当今的大语言模型(预)训练和研究流程通常会分配大量计算资源用于大规模消融研究。尽管这些消融研究的计算成本很高,但现有的开源框架提供的工具有限,通常迫使研究人员自行编写包装器和脚本。我们提出了Modalities,一个端到端的PyTorch原生框架,从两个角度整合数据驱动的大语言模型研究与大规模模型训练。首先,通过整合最先进的并行化策略,它使在万亿token和十亿参数规模上实现高效的预训练和系统的消融研究成为可能。其次,Modalities采用模块化设计,具有声明式、自包含的配置,使可重复性和可扩展性达到了现有大语言模型训练框架难以实现的水平。
英文摘要:
Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of these ablations, existing open-source frameworks provide limited tooling for these experiments, often forcing researchers to write their own wrappers and scripts. We propose Modalities, an end-to-end PyTorch-native framework that integrates data-driven LLM research with large-scale model training from two angles. Firstly, by integrating state-of-the-art parallelization strategies, it enables both efficient pretraining and systematic ablations at trillion-token and billion-parameter scale. Secondly, Modalities adopts modular design with declarative, self-contained configuration, enabling reproducibility and extensibility levels that are difficult to achieve out-of-the-box with existing LLM training frameworks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。