Nemotron 3 Nano:开放、高效专家混合混合Mamba-Transformer模型用于代理推理
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
AI总结:
Nemotron 3 Nano是一种高效混合Mamba-Transformer模型,通过预训练和微调在代理推理任务中实现更高准确性和吞吐量。
摘要:
我们介绍了Nemotron 3 Nano 30B-A3B,一种混合Mamba-Transformer语言模型。Nemotron 3 Nano在25万亿个文本标记上进行了预训练,包括比Nemotron 2多超过3万亿个新唯一标记,随后进行监督微调和大规模RL在多样化的环境中。Nemotron 3 Nano在每次前向传递中激活的参数少于一半,同时比我们之前的Nemotron 2 Nano实现了更高的准确性。它在类似规模的开放模型如GPT-OSS-20B和Qwen3-30B-A3B-Thinking-2507上实现了高达3.3倍的推理吞吐量,同时在流行的基准测试中也更准确。Nemotron 3 Nano展示了增强的代理、推理和聊天能力,并支持高达100万个标记的上下文长度。我们发布了预训练的Nemotron 3 Nano 30B-A3B基础和后训练的Nemotron 3 Nano 30B-A3B检查点在Hugging Face上。
英文摘要:
We present Nemotron 3 Nano 30B-A3B, a Mixture-of-Experts hybrid Mamba-Transformer language model. Nemotron 3 Nano was pretrained on 25 trillion text tokens, including more than 3 trillion new unique tokens over Nemotron 2, followed by supervised fine tuning and large-scale RL on diverse environments. Nemotron 3 Nano achieves better accuracy than our previous generation Nemotron 2 Nano while activating less than half of the parameters per forward pass. It achieves up to 3.3x higher inference throughput than similarly-sized open models like GPT-OSS-20B and Qwen3-30B-A3B-Thinking-2507, while also being more accurate on popular benchmarks. Nemotron 3 Nano demonstrates enhanced agentic, reasoning, and chat abilities and supports context lengths up to 1M tokens. We release both our pretrained Nemotron 3 Nano 30B-A3B Base and post-trained Nemotron 3 Nano 30B-A3B checkpoints on Hugging Face.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。