arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11937cs.LG
收藏

扩展谜题以用于专家混合推理模型的推理,应用于GPT-OSS加速

Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration

Akhiad Bercovich, Nir Ailon, Vladimir Anisimov, Tomer Asida, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Roi Koren, Ita… 展开作者

Akhiad Bercovich, Nir Ailon, Vladimir Anisimov, Tomer Asida, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Roi Koren, Itay Levy, Zach Moshe, Pavlo Molchanov, Najeeb Nabwani, Mostofa Patwary, Omri Puny, Tomer Ronen, Itamar Schen, Elad Segal, Ido Shahaf, Oren Tropp, Ran Zilberstein, Ran El-Yaniv

AI总结:

本文通过扩展Puzzle框架优化GPT-OSS模型,结合MoE剪枝、注意力替换、量化和强化学习提升推理效率,实现部署优化并提高请求级效率。

摘要:

面向推理的LLM通过生成更长的推理轨迹提高回答质量,但额外的token显著增加服务成本,推动了推理优化。我们扩展并应用Puzzle,一种后训练神经网络架构搜索(NAS)框架,到gpt-oss-120B以生成gpt-oss-puzzle-88B,一种部署优化的衍生模型。我们的方法结合异构MoE专家剪枝、选择性替换全上下文注意力为窗口注意力、FP8 KV缓存量化与校准尺度,以及后训练强化学习以恢复准确性,同时保持低生成长度。在8XH100节点上,我们实现了长上下文和短上下文设置下的1.63X和1.22X吞吐量提升。gpt-oss-puzzle-88B还在单个NVIDIA H100 GPU上实现了2.82X的吞吐量提升。然而,由于token数量可能随着推理努力和模型变体而变化,每token吞吐量(tok/s)和延迟(ms/token)不必然导致端到端加速:2X吞吐量提升如果轨迹增长2X则会被抵消。相反,吞吐量提升可以用于更多推理token以提高准确性;因此,我们倡导请求级效率度量,将吞吐量归一化为生成的token,并在推理努力中追踪准确性-速度前沿。我们证明gpt-oss-puzzle-88B在前沿上优于gpt-oss-120B,最高达到1.29X的请求级效率提升。在各种基准测试中,gpt-oss-puzzle-88B在推理努力中匹配或略微超过父模型的套件平均准确性,保留率范围从100.8%(高)到108.2%(低),表明后训练架构搜索可以显著降低推理成本而不牺牲质量。

英文摘要:

Reasoning-focused LLMs improve answer quality by generating longer reasoning traces, but the additional tokens dramatically increase serving cost, motivating inference optimization. We extend and apply Puzzle, a post-training neural architecture search (NAS) framework, to gpt-oss-120B to produce gpt-oss-puzzle-88B, a deployment-optimized derivative. Our approach combines heterogeneous MoE expert pruning, selective replacement of full-context attention with window attention, FP8 KV-cache quantization with calibrated scales, and post-training reinforcement learning to recover accuracy, while maintaining low generation length. In terms of per-token speeds, on an 8XH100 node we achieve 1.63X and 1.22X throughput speedups in long-context and short-context settings, respectively. gpt-oss-puzzle-88B also delivers throughput speedups of 2.82X on a single NVIDIA H100 GPU. However, because token counts can change with reasoning effort and model variants, per-token throughput (tok/s) and latency (ms/token) do not necessarily lead to end-to-end speedups: a 2X throughput gain is erased if traces grow 2X. Conversely, throughput gains can be spent on more reasoning tokens to improve accuracy; we therefore advocate request-level efficiency metrics that normalize throughput by tokens generated and trace an accuracy--speed frontier across reasoning efforts. We show that gpt-oss-puzzle-88B improves over gpt-oss-120B along the entire frontier, delivering up to 1.29X higher request-level efficiency. Across various benchmarks, gpt-oss-puzzle-88B matches or slightly exceeds the parent on suite-average accuracy across reasoning efforts, with retention ranging from 100.8% (high) to 108.2% (low), showing that post-training architecture search can substantially reduce inference costs without sacrificing quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑