arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12635cs.CLcs.AIcs.LG
收藏

释放Ascend NPUs上的低比特推理:HiFloat格式的全面评估

Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats

Huawei Technologies Co., Ltd(华为技术有限公司)

Pengxiang Zhao, Hui-Ling Zhen, Xing Li, Han Bao, Weizhe Lin, Zhiyuan Yang, Manyi Zhang, Yuanyong Luo, Ziwei Yu, Xin Wang, Mingxuan Yuan, Xianzhi Yu, Zhenhua Dong

AI总结:

本文评估了HiFloat格式在Ascend NPUs上的低比特推理性能,揭示了不同格式在精度、效率及兼容性方面的关键差异。

摘要:

随着大语言模型的扩展,像MXFP和NVFP4这样的低比特浮点格式为精度和效率提供了新的机会。在本工作中,我们评估了HiFloat(HiF8和HiF4),一种针对Ascend NPUs设计的格式家族。通过在权重-激活和KV缓存任务上的严格比较,我们提供了三个关键见解:(1)INT8适用于窄范围数据,而浮点格式在高变异性数据中表现更佳;(2)在4比特范围内,HiF4的分层扩展防止了整数格式中出现的精度崩溃;(3)HiFloat完全兼容最先进的后训练量化框架。总体而言,HiFloat为在NPUs上实现高效率的大语言模型推理提供了解决方案。

英文摘要:

As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑