可逆基础：通过状态保持缩放训练120B稀疏MoE
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
AI总结：
本文报告在单台8-GPU节点上端到端训练千亿参数稀疏混合专家模型，通过可逆循环、状态保持增长和单节点经济学三大原则，实现从密集种子到120B模型的四阶段扩展。
摘要：
本文报告在单台八GPU节点上端到端训练千亿参数稀疏混合专家模型。LightningLM 0.1V是一个循环骨干语言模型家族，通过四个阶段从小型密集种子扩展，经过5B和9B混合专家，最终达到120B模型，具有460个路由专家，采用top-12路由。每个更大模型从小模型的训练权重增长而来；活跃参数从密集种子的1.78B单调增加到120B时的5.93B（约占存储的118.67B的5%）。整个谱系在单节点上运行，较大阶段在8K上下文中，达到120B规模时发布的训练损失为1.78。这是一份系统和经验报告，围绕三个原则组织。可逆性：可逆循环栈在反向传播中重建激活而非存储它们，使激活内存随模型增长保持平坦。状态保持增长：每次扩展（密集到MoE、浅到深、少专家到多专家）都作为可重现原则给出，并附有错误导致的失败案例；若干失败是无声的。单节点经济学：120B通过TQP训练，这是一种量化基础专家权重和训练低秩适配器的策略，将优化器状态承载于2.26B适配器参数而非路由专家中的100B+，将专家路径优化器状态减少约45倍。新颖之处在于已知原语的集成，而非任何孤立原语：一个在单节点上端到端运行的成长谱系，以从业者级别记录，并以每个领域的留出损失作为证据，表明目标能力（多语言印度能力、代码）是通过构造学习的。模型家族、分词器和训练代码已发布。
英文摘要：
This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end. LightningLM 0.1V is a recurrence-backbone language model family grown in four stages from a small dense seed, through a 5B and a 9B mixture of experts, to a 120B model with 460 routed experts under top-12 routing. Each larger model is grown from the trained weights of the smaller one; active parameters rise monotonically from 1.78B at the dense seed to 5.93B at 120B (about 5% of the 118.67B stored). The full lineage runs on single nodes, the larger stages at 8K context, reaching a released training loss of 1.78 at 120B scale. This is a systems and experience report. It is organized around three disciplines. Reversibility: a reversible recurrence stack reconstructs activations in the backward pass instead of storing them, holding activation memory flat as the model grows. State-preserving growth: each expansion (dense to MoE, shallow to deep, few experts to many) is given as a reproducible principle paired with the failure that results from getting it wrong; several failures are silent. Single-node economics: the 120B trains through TQP, a strategy of quantized base expert weights and trained low-rank adapters that carries optimizer state on 2.26B adapter parameters rather than 100B+ resident in routed experts, cutting expert-path optimizer state by a factor of ~45. What is new is the integration of known primitives, not any primitive in isolation: one grown lineage running end to end on a single node, documented at practitioner level, with per-domain held-out loss as evidence that targeted capabilities (multilingual Indic competence, code) were learned by construction. Model family, tokenizer, and training code are released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
