We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16. To our knowledge, HiFA4 is the first Ascend-HIF4-targeted design of this kind evaluated on standard NLP benchmarks.
HiFA4 combines two mechanisms. Smooth-QK applies a calibration-static per-channel equivalent rescaling to Q and K after RoPE, transferring quantization difficulty from K to Q without per-tile online reduction at inference. P-Reordering accumulates the softmax normalizer from the same quantized attention weights P_hat used in the PV GEMM, rather than from a higher-precision reconstruction. We show that this inconsistent formulation introduces a coherent output-scaling error, and validate the effect on a Qwen3-8B Layer-0 MMLU trace, where all 3.6M measured attention tiles exhibit net probability-mass loss with median epsilon_bar = -0.064. P-Reordering also allows the normalizer to be fused into the PV Cube GEMM.
Across five LLMs, HiFA4 reduces quantization-induced decision drift. On Qwen3-8B, it recovers 37.5% of the accuracy gap introduced by direct HIF4 quantization, narrows the sample-weighted accuracy loss from 1.12 pp to 0.70 pp, reduces BF16-inconsistent MMLU predictions from 16.3% to 8.2%, and cuts MMLU accuracy regressions by 57% (1071 to 465). On Gemma2-9B, mild smoothing keeps HiFA4 within 0.7 pp of BF16 while reducing MMLU regressions by 27%. On LLaMA3.1-8B, Mistral-7B, and Phi-4B, where Smooth-QK is disabled, P-Reordering with the adopted Q-Mean auxiliary still reduces full-set MMLU regressions by 41-52%. A preliminary instruction-scheduling analysis projects a 35.4% critical-path latency reduction relative to BF16 by fusing the softmax normalizer into the PV Cube GEMM; on-hardware validation is left to future work.
An Empirical Study of openPangu Quantization on Ascend NPUs
State Key Laboratory of Complex & Critical Software Environment, Beihang University(北京航空航天大学复杂关键软件环境国家重点实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院)
STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
AMD Research and Advanced Development (RAD)(AMD研究与先进开发);Integrated Systems Laboratory (IIS)(集成系统实验室);ETH Zürich(苏黎世联邦理工学院);Department of Electrical, Electronic and Information Engineering (DEI)(电气电子信息工程系);University of Bologna(博洛尼亚大学)
ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Technologies Co., Ltd.(华为技术有限公司)
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
Huawei(华为);Tsinghua University(清华大学);Shanghai AI Lab(上海人工智能实验室)
KLineage: Recovering the Missing When of Kernel Optimization by Deoptimizing Experts
SKLP, Institute of Computing Technology, Chinese Academy of Sciences(SKLP,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);University of New South Wales(新南威尔士大学)
AI-RAN on NPUs: Baseband Processing Without Baseband Chips
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Intelligent Software and Engineering, Nanjing University (Suzhou Campus)(南京大学智能软件与工程学院(苏州校区));National Key Laboratory of Science and Technology on Communications, University of Electronic Science and Technology of China(中国电子科学技术研究院通信科学技术国家重点实验室)