TurboMamba:借助词汇记忆实现更快推理、更快训练与更优预测
TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory
AI总结:
TurboMamba是一种语言模型架构,通过全局Mamba-3与小型Transformer结合三项轻量改进,在参数规模相近的对比中,实现了更低预测损失、更高训练吞吐量与更快推理速度。
摘要:
我们提出TurboMamba,一种可实现更快推理、更快训练与更优token预测的语言模型架构。全局Mamba-3模型每完成一个token块运行一次,小型Transformer利用该全局上下文与块内更早的token预测下一个token。三项轻量改进提升局部预测能力:直接访问前一个块的最后一个token、对下一个token得分进行小型修正、结合近期token频率的学习混合机制。我们在公开FineWeb-Edu数据集上对比了12种小规模实现,参数规模相近且训练预算上限一致,基线模型包括Transformer、Mamba-3、Jamba和Zamba,其中Jamba与Zamba版本用Mamba-3替换了原有的Mamba模块。TurboMamba在所有12种模型中实现了最低的平均测试预测损失(负对数似然,NLL),其平均NLL比Mamba-3低9.4%,比Jamba版本低7.3%,比Zamba版本低18.3%;端到端训练吞吐量是Mamba-3的2.60倍、Jamba版本的2.81倍、Zamba版本的4.82倍。我们针对一批包含32个序列的 workload 测量端到端批量延迟,每个序列有256个token的提示和128个生成token,在此 workload 下,TurboMamba比Mamba-3提速3.12倍、比Jamba版本提速3.53倍、比Zamba版本提速6.21倍。
英文摘要:
We introduce TurboMamba, a language-model architecture for faster inference, faster training, and better token prediction. A global Mamba-3 model runs once per completed token block. A small Transformer uses this global context and earlier tokens within the block to predict the next token. Three lightweight additions improve local predictions: direct access to the previous block's last token, a small correction to next-token scores, and a learned mixture with recent-token frequencies. We compare twelve small-scale implementations on public FineWeb-Edu, with comparable parameter counts and shared training-budget caps. The baselines include Transformer, Mamba-3, Jamba, and Zamba. The Jamba and Zamba versions use Mamba-3 in place of their original Mamba modules. TurboMamba achieves the lowest mean test prediction loss (negative log-likelihood, NLL) among all twelve models. Its mean NLL is 9.4% lower than that of Mamba-3, 7.3% lower than that of the Jamba version, and 18.3% lower than that of the Zamba version. End-to-end training throughput is 2.60x that of Mamba-3, 2.81x that of the Jamba version, and 4.82x that of the Zamba version. We measure end-to-end batch latency for a batch of 32 sequences, each with a 256-token prompt and 128 generated tokens. On this workload, TurboMamba achieves a 3.12x speedup over Mamba-3, a 3.53x speedup over the Jamba version, and a 6.21x speedup over the Zamba version.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。