arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.23367cs.LGcs.AI
收藏

在Atlas A2上对OpenPangu模型进行训练后量化以实现高效部署

Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2

School of Computer Science and Technology, Tianjin University(计算机科学与技术学院,天津大学)

Yilun Luo, Huaqing Zheng, Haoqian Meng, Wenyuan Liu, Peng Zhang

AI总结:

本文提出在Atlas A2上通过低比特量化提升OpenPangu模型的推理效率,实现高效CoT推理并保持高精度。

摘要:

华为的openPangu-Embedded-1B和openPangu-Embedded-7B是为在Ascend NPUs上高效部署而设计的openPangu大语言模型变种。7B变种支持三种不同的链式推理(CoT)范式,即slow_think、auto_think和no_think,而1B变种仅在no_think模式下运行,该模式采用压缩推理以提高效率。尽管CoT推理增强了能力,但生成扩展推理轨迹引入了显著的内存和延迟开销,这对在Ascend NPUs上的实际部署构成了挑战。本文通过利用低比特量化来解决这些计算限制,将FP16计算转换为更高效的整数运算。我们引入了一个统一的低比特推理框架,支持INT8(W8A8)和W4A8量化,特别针对Atlas A2上的openPangu-Embedded模型进行了优化。我们在代码生成基准(HumanEval和MBPP)上的全面评估证明了这种方法的有效性。INT8量化在Atlas A2上实现了1.5倍的prefill加速,同时保持超过90%的FP16基线精度。此外,W4A8量化显著减少了内存消耗,尽管在精度上有一个适度的权衡。这些发现共同表明,低比特量化有效促进了在Ascend NPUs上的高效CoT推理,同时保持高模型保真度。

英文摘要:

Huawei's openPangu-Embedded-1B and openPangu-Embedded-7B are variants of the openPangu large language model, designed for efficient deployment on Ascend NPUs. The 7B variant supports three distinct Chain-of-Thought (CoT) reasoning paradigms, namely slow_think, auto_think, and no_think, while the 1B variant operates exclusively in the no_think mode, which employs condensed reasoning for higher efficiency. Although CoT reasoning enhances capability, the generation of extended reasoning traces introduces substantial memory and latency overheads, posing challenges for practical deployment on Ascend NPUs. This paper addresses these computational constraints by leveraging low-bit quantization, which transforms FP16 computations into more efficient integer arithmetic. We introduce a unified low-bit inference framework, supporting INT8 (W8A8) and W4A8 quantization, specifically optimized for openPangu-Embedded models on the Atlas A2. Our comprehensive evaluation on code generation benchmarks (HumanEval and MBPP) demonstrates the efficacy of this approach. INT8 quantization consistently preserves over 90\% of the FP16 baseline accuracy and achieves a 1.5x prefill speedup on the Atlas A2. Furthermore, W4A8 quantization significantly reduces memory consumption, albeit with a moderate trade-off in accuracy. These findings collectively indicate that low-bit quantization effectively facilitates efficient CoT reasoning on Ascend NPUs, maintaining high model fidelity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑