arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.08313cs.CLcs.CV
收藏

MiniMax-01: 借助闪电注意力(Lightning Attention)扩展基础模型

MiniMax-01: Scaling Foundation Models with Lightning Attention

MiniMax
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang,… 展开作者

MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, Zijia Wu

AI总结:

本研究推出MiniMax-01系列文本与视觉语言模型,核心采用闪电注意力结合MoE架构及优化并行策略,实现百万级token长上下文高效训练推理,性能比肩前沿模型且上下文窗口长20-32倍,已开源发布。

摘要:

我们推出MiniMax-01系列,包含MiniMax-Text-01和MiniMax-VL-01,其性能可与顶级模型比肩,同时在处理长上下文方面具备更优异的能力。该系列的核心在于闪电注意力(lightning attention)及其高效扩展机制。为最大化计算效能,我们将其与混合专家模型(Mixture of Experts, MoE)相结合,构建了一个拥有32个专家、总参数量达4560亿的模型,其中每个token的激活参数量为459亿。我们针对MoE和闪电注意力开发了优化的并行策略以及高效的计算-通信重叠技术。这一方案使我们能够在数百万token的上下文长度下,对数千亿参数规模的模型进行高效训练与推理。MiniMax-Text-01的上下文窗口在训练阶段最高可达100万token,在推理阶段可外推至400万token,且成本可控。我们的视觉语言模型MiniMax-VL-01是通过使用5120亿视觉语言token进行持续训练构建而成。在标准基准与内部基准上开展的实验表明,我们的模型性能与GPT-4o、Claude-3.5-Sonnet等前沿模型相当,同时提供了20至32倍更长的上下文窗口。我们已在https://github.com/MiniMax-AI公开发布MiniMax-01。

英文摘要:

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑