arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.17693cs.LGcs.AIcs.CL
收藏

在Ascend NPU上对推理LLM的选定PTQ基线进行案例研究

A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU

School of Mathematics and Statistics, Wuhan University(武汉大学数学与统计学院);Huawei(华为公司)

Yuchen Luo, Fangyue Zhu, Ruining Zhou, Mingzhe Huang, Jian Zhu, Fanyu Fan, Wei Shao

AI总结:

本文研究了在Ascend NPU上不同PTQ方法对推理LLM的影响,发现4位权重量化在大模型中可行,但激进方案存在稳定性问题,而8位量化更稳定,实际部署中动态量化开销限制了加速效果。

摘要:

后训练量化(PTQ)对于高效模型部署至关重要,但其在Ascend NPU上的有效性相较于GPU架构仍研究不足。本文展示了代表性PTQ基线应用于推理导向模型如DeepSeek-R1-Distill-Qwen系列(1.5B/7B/14B)和QwQ-32B的案例研究。我们评估了四种不同的算法,包括AWQ、GPTQ、SmoothQuant和FlatQuant,以涵盖从权重-only压缩到先进旋转方法的整个范围。我们的实证结果揭示了显著的平台敏感性。虽然4位权重-only量化对较大模型可行,但激进的4位权重-激活方案在NPU上因层间校准不稳定而导致长上下文推理任务中的逻辑崩溃。相反,标准8位量化保持数值稳定。此外,实际的INT8部署显示,尽管优化内核减少了延迟,动态量化开销目前限制了端到端加速。这些发现为在Ascend NPU上部署量化推理模型的可行性和限制提供了实用参考。

英文摘要:

Post-Training Quantization (PTQ) is crucial for efficient model deployment, yet its effectiveness on Ascend NPU remains under-explored compared to GPU architectures. This paper presents a case study of representative PTQ baselines applied to reasoning-oriented models such as DeepSeek-R1-Distill-Qwen series (1.5B/7B/14B) and QwQ-32B. We evaluate four distinct algorithms, including AWQ, GPTQ, SmoothQuant, and FlatQuant, to cover the spectrum from weight-only compression to advanced rotation-based methods. Our empirical results reveal significant platform sensitivity. While 4-bit weight-only quantization proves viable for larger models, aggressive 4-bit weight-activation schemes suffer from layer-wise calibration instability on the NPU, leading to logic collapse in long-context reasoning tasks. Conversely, standard 8-bit quantization remains numerically stable. Furthermore, a real-world INT8 deployment demonstrates that although optimized kernels reduce latency, dynamic quantization overheads currently limit end-to-end acceleration. These findings offer a practical reference for the feasibility and limitations of deploying quantized reasoning models on Ascend NPU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑